2024-04-26T01:09:23.503 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T01:09:23.503 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000 --verbose --description rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} -- /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/orig.config.yaml 2024-04-26T01:09:23.543 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T01:09:23.618 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000 branch: squid description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: ceph-qa@ceph.com first_in_suite: false job_id: '7674000' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 9886 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 30270/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b22e2ebdeb24376882b7bda2a7329c8cccc2276a targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcxhBjkX8tb5Wy+F24m7wrrFVlxtgpld3aT4x0qj8ih5k4DFrcx/2f3mQfwLvaBLPzmr4upe0vzs1AXF9NoLSY= smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN568PHByclQZBg9C/JgU/9kWxRTDJyKuOAnv56HtX4obqVy69BIZru/AWFJlpgZy5iA6/0FrdSrJISHEySIERA= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFamGPdQPSnJICWOC9Wbj/ORtCpsQNWbZXU5J4LP6zXmQ8y1hvC1fgtmK7R3NKznhYFsRsap6YXnXDXX4QuBRRo= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLOab7Xe2nDHX4inIWcwqEaS+VU9btv4yUIgQcapfkkr307sEivv3ZNIFwOI6dPgw1AUzSFVCH6WmAQXxXCRrxQ= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_21:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T01:09:23.618 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa; will attempt to use it 2024-04-26T01:09:23.619 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa/tasks 2024-04-26T01:09:23.619 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T01:09:23.620 INFO:teuthology.task.internal:Checking packages... 2024-04-26T01:09:23.651 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-26T01:09:23.651 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T01:09:23.651 INFO:teuthology.packaging:ref: None 2024-04-26T01:09:23.651 INFO:teuthology.packaging:tag: None 2024-04-26T01:09:23.651 INFO:teuthology.packaging:branch: squid 2024-04-26T01:09:23.651 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:09:23.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-26T01:09:23.940 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2447-gb22e2ebd-1jammy 2024-04-26T01:09:23.941 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T01:09:24.125 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T01:09:24.125 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T01:09:24.147 INFO:teuthology.task.internal:Saving configuration 2024-04-26T01:09:24.163 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T01:09:24.168 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T01:09:24.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:03:48.641072', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN/6X6XVpZ9jTUHpYkrBoXeOyef8JXs+f4fsIiY98oSP'} 2024-04-26T01:09:24.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:03:48.639874', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAaaClIBClwYoeguoTb/MX5wKjnkNaTY//7Rn+pzYWAg'} 2024-04-26T01:09:24.241 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:03:48.638358', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuVWueu8xvYuGTdfHbGtOv2khs2SVQ4emk0jApm6LWB'} 2024-04-26T01:09:24.261 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 01:03:48.642430', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHzt8XKGmCfgp8ExZ99CutufS8+kF+ZY4ufr1IiFAj1D'} 2024-04-26T01:09:24.261 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T01:09:24.268 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-04-26T01:09:24.268 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-04-26T01:09:24.268 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-04-26T01:09:24.268 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-04-26T01:09:24.268 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T01:09:24.401 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6d0a0ad280>, signals=[15]) 2024-04-26T01:09:24.402 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T01:09:24.408 INFO:teuthology.task.internal:Opening connections... 2024-04-26T01:09:24.409 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:09:24.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:09:24.505 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:09:24.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:09:24.589 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:09:24.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:09:24.667 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:09:24.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:09:24.746 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T01:09:24.753 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-04-26T01:09:24.759 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-04-26T01:09:24.759 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="22.04" 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=jammy 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2024-04-26T01:09:24.808 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T01:09:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T01:09:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T01:09:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T01:09:24.809 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=jammy 2024-04-26T01:09:24.809 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-04-26T01:09:24.826 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-04-26T01:09:24.832 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-04-26T01:09:24.832 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-04-26T01:09:24.880 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Ubuntu" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="22.04" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_CODENAME=jammy 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:ID=ubuntu 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE=debian 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T01:09:24.881 INFO:teuthology.orchestra.run.smithi045.stdout:UBUNTU_CODENAME=jammy 2024-04-26T01:09:24.881 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-04-26T01:09:24.901 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-04-26T01:09:24.907 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-04-26T01:09:24.908 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="Ubuntu" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="22.04" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_CODENAME=jammy 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:ID=ubuntu 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE=debian 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T01:09:24.953 INFO:teuthology.orchestra.run.smithi071.stdout:UBUNTU_CODENAME=jammy 2024-04-26T01:09:24.954 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-04-26T01:09:24.967 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-04-26T01:09:24.973 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-04-26T01:09:24.973 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-04-26T01:09:25.022 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T01:09:25.022 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Ubuntu" 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="22.04" 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_CODENAME=jammy 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:ID=ubuntu 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE=debian 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T01:09:25.023 INFO:teuthology.orchestra.run.smithi181.stdout:UBUNTU_CODENAME=jammy 2024-04-26T01:09:25.023 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-04-26T01:09:25.043 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T01:09:25.055 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T01:09:25.060 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T01:09:25.061 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:09:25.063 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:09:25.066 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:09:25.068 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-04-26T01:09:25.073 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T01:09:25.079 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T01:09:25.079 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:09:25.113 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:09:25.117 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:09:25.119 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-04-26T01:09:25.194 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T01:09:25.292 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T01:09:25.308 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T01:09:25.308 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T01:09:25.308 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:09:25.309 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:09:25.309 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:09:25.310 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T01:09:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:09:25.314 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-04-26T01:09:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:09:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:09:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:09:25.318 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-04-26T01:09:25.318 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-04-26T01:09:25.318 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-26T01:09:25.362 INFO:teuthology.orchestra.run.smithi071.stdout:5.15.0-56-generic 2024-04-26T01:09:25.362 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.15.0-56-generic 2024-04-26T01:09:25.363 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2024-04-26T01:09:25.365 INFO:teuthology.orchestra.run.smithi037.stdout:5.15.0-56-generic 2024-04-26T01:09:25.365 INFO:teuthology.orchestra.run.smithi045.stdout:5.15.0-56-generic 2024-04-26T01:09:25.366 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.15.0-56-generic 2024-04-26T01:09:25.366 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-04-26T01:09:25.366 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.15.0-56-generic 2024-04-26T01:09:25.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2024-04-26T01:09:25.368 INFO:teuthology.orchestra.run.smithi181.stdout:5.15.0-56-generic 2024-04-26T01:09:25.369 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.15.0-56-generic 2024-04-26T01:09:25.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get clean 2024-04-26T01:09:25.489 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-04-26T01:09:25.503 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-04-26T01:09:25.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get update 2024-04-26T01:09:25.514 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2024-04-26T01:09:25.642 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T01:09:25.664 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:09:25.691 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T01:09:25.692 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:09:25.692 INFO:teuthology.orchestra.run.smithi181.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T01:09:25.692 INFO:teuthology.orchestra.run.smithi181.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T01:09:25.741 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:09:25.770 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T01:09:25.777 INFO:teuthology.orchestra.run.smithi181.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T01:09:25.823 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T01:09:25.867 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T01:09:25.906 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T01:09:25.939 INFO:teuthology.orchestra.run.smithi181.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T01:09:25.989 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T01:09:26.007 INFO:teuthology.orchestra.run.smithi181.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T01:09:26.020 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T01:09:26.032 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T01:09:26.039 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T01:09:26.040 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T01:09:26.041 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T01:09:26.047 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T01:09:26.048 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T01:09:26.049 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T01:09:26.070 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T01:09:26.070 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T01:09:26.075 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.075 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T01:09:26.082 INFO:teuthology.orchestra.run.smithi181.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T01:09:26.088 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T01:09:26.093 INFO:teuthology.orchestra.run.smithi181.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T01:09:26.094 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.094 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T01:09:26.097 INFO:teuthology.orchestra.run.smithi181.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T01:09:26.097 INFO:teuthology.orchestra.run.smithi181.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T01:09:26.100 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T01:09:26.108 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T01:09:26.114 INFO:teuthology.orchestra.run.smithi045.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T01:09:26.120 INFO:teuthology.orchestra.run.smithi045.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T01:09:26.124 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T01:09:26.125 INFO:teuthology.orchestra.run.smithi045.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T01:09:26.126 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T01:09:26.126 INFO:teuthology.orchestra.run.smithi045.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T01:09:26.127 INFO:teuthology.orchestra.run.smithi045.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T01:09:26.131 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T01:09:26.134 INFO:teuthology.orchestra.run.smithi181.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T01:09:26.137 INFO:teuthology.orchestra.run.smithi181.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T01:09:26.137 INFO:teuthology.orchestra.run.smithi181.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T01:09:26.141 INFO:teuthology.orchestra.run.smithi181.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.141 INFO:teuthology.orchestra.run.smithi181.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T01:09:26.154 INFO:teuthology.orchestra.run.smithi181.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T01:09:26.159 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T01:09:26.159 INFO:teuthology.orchestra.run.smithi045.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T01:09:26.160 INFO:teuthology.orchestra.run.smithi045.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T01:09:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T01:09:26.160 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T01:09:26.160 INFO:teuthology.orchestra.run.smithi045.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T01:09:26.162 INFO:teuthology.orchestra.run.smithi045.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T01:09:26.162 INFO:teuthology.orchestra.run.smithi181.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T01:09:26.163 INFO:teuthology.orchestra.run.smithi045.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T01:09:26.163 INFO:teuthology.orchestra.run.smithi045.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T01:09:26.163 INFO:teuthology.orchestra.run.smithi045.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T01:09:26.164 INFO:teuthology.orchestra.run.smithi181.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T01:09:26.167 INFO:teuthology.orchestra.run.smithi045.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T01:09:26.181 INFO:teuthology.orchestra.run.smithi045.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T01:09:26.182 INFO:teuthology.orchestra.run.smithi045.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T01:09:26.188 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T01:09:26.194 INFO:teuthology.orchestra.run.smithi181.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T01:09:26.194 INFO:teuthology.orchestra.run.smithi181.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T01:09:26.194 INFO:teuthology.orchestra.run.smithi181.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T01:09:26.194 INFO:teuthology.orchestra.run.smithi181.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T01:09:26.194 INFO:teuthology.orchestra.run.smithi181.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T01:09:26.200 INFO:teuthology.orchestra.run.smithi181.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T01:09:26.201 INFO:teuthology.orchestra.run.smithi181.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T01:09:26.202 INFO:teuthology.orchestra.run.smithi181.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T01:09:26.204 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:09:26.204 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T01:09:26.204 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T01:09:26.232 INFO:teuthology.orchestra.run.smithi181.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T01:09:26.236 INFO:teuthology.orchestra.run.smithi181.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.236 INFO:teuthology.orchestra.run.smithi181.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T01:09:26.244 INFO:teuthology.orchestra.run.smithi181.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T01:09:26.254 INFO:teuthology.orchestra.run.smithi181.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T01:09:26.256 INFO:teuthology.orchestra.run.smithi181.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T01:09:26.256 INFO:teuthology.orchestra.run.smithi181.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T01:09:26.257 INFO:teuthology.orchestra.run.smithi181.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T01:09:26.257 INFO:teuthology.orchestra.run.smithi181.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T01:09:26.266 INFO:teuthology.orchestra.run.smithi181.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T01:09:26.266 INFO:teuthology.orchestra.run.smithi181.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T01:09:26.267 INFO:teuthology.orchestra.run.smithi181.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T01:09:26.277 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T01:09:26.278 INFO:teuthology.orchestra.run.smithi181.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T01:09:26.279 INFO:teuthology.orchestra.run.smithi181.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T01:09:26.279 INFO:teuthology.orchestra.run.smithi181.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T01:09:26.280 INFO:teuthology.orchestra.run.smithi181.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T01:09:26.280 INFO:teuthology.orchestra.run.smithi181.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T01:09:26.287 INFO:teuthology.orchestra.run.smithi181.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T01:09:26.323 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T01:09:26.339 INFO:teuthology.orchestra.run.smithi045.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T01:09:26.414 INFO:teuthology.orchestra.run.smithi045.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T01:09:26.444 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T01:09:26.458 INFO:teuthology.orchestra.run.smithi045.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T01:09:26.464 INFO:teuthology.orchestra.run.smithi045.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T01:09:26.465 INFO:teuthology.orchestra.run.smithi045.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T01:09:26.466 INFO:teuthology.orchestra.run.smithi045.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T01:09:26.497 INFO:teuthology.orchestra.run.smithi045.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T01:09:26.501 INFO:teuthology.orchestra.run.smithi045.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.501 INFO:teuthology.orchestra.run.smithi045.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T01:09:26.508 INFO:teuthology.orchestra.run.smithi045.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T01:09:26.515 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T01:09:26.516 INFO:teuthology.orchestra.run.smithi045.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T01:09:26.518 INFO:teuthology.orchestra.run.smithi045.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T01:09:26.518 INFO:teuthology.orchestra.run.smithi045.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T01:09:26.519 INFO:teuthology.orchestra.run.smithi045.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T01:09:26.519 INFO:teuthology.orchestra.run.smithi045.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T01:09:26.523 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T01:09:26.526 INFO:teuthology.orchestra.run.smithi045.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T01:09:26.533 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T01:09:26.578 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T01:09:26.582 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T01:09:26.582 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T01:09:26.605 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T01:09:26.606 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T01:09:26.610 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.610 INFO:teuthology.orchestra.run.smithi071.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T01:09:26.619 INFO:teuthology.orchestra.run.smithi071.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T01:09:26.621 INFO:teuthology.orchestra.run.smithi071.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T01:09:26.626 INFO:teuthology.orchestra.run.smithi071.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T01:09:26.628 INFO:teuthology.orchestra.run.smithi071.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T01:09:26.649 INFO:teuthology.orchestra.run.smithi071.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T01:09:26.649 INFO:teuthology.orchestra.run.smithi071.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T01:09:26.649 INFO:teuthology.orchestra.run.smithi071.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T01:09:26.649 INFO:teuthology.orchestra.run.smithi071.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T01:09:26.649 INFO:teuthology.orchestra.run.smithi071.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T01:09:26.654 INFO:teuthology.orchestra.run.smithi071.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T01:09:26.655 INFO:teuthology.orchestra.run.smithi071.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T01:09:26.673 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T01:09:26.685 INFO:teuthology.orchestra.run.smithi071.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T01:09:26.686 INFO:teuthology.orchestra.run.smithi071.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T01:09:26.687 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T01:09:26.688 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T01:09:26.691 INFO:teuthology.orchestra.run.smithi071.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.691 INFO:teuthology.orchestra.run.smithi071.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T01:09:26.700 INFO:teuthology.orchestra.run.smithi071.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T01:09:26.711 INFO:teuthology.orchestra.run.smithi071.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T01:09:26.713 INFO:teuthology.orchestra.run.smithi071.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T01:09:26.713 INFO:teuthology.orchestra.run.smithi071.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T01:09:26.714 INFO:teuthology.orchestra.run.smithi071.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T01:09:26.718 INFO:teuthology.orchestra.run.smithi071.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T01:09:26.719 INFO:teuthology.orchestra.run.smithi071.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T01:09:26.719 INFO:teuthology.orchestra.run.smithi071.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T01:09:26.719 INFO:teuthology.orchestra.run.smithi071.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T01:09:26.736 INFO:teuthology.orchestra.run.smithi071.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T01:09:26.736 INFO:teuthology.orchestra.run.smithi071.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T01:09:26.736 INFO:teuthology.orchestra.run.smithi071.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T01:09:26.736 INFO:teuthology.orchestra.run.smithi071.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T01:09:26.737 INFO:teuthology.orchestra.run.smithi071.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T01:09:26.739 INFO:teuthology.orchestra.run.smithi071.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T01:09:26.751 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T01:09:26.752 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T01:09:26.759 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T01:09:26.923 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T01:09:27.562 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T01:09:27.599 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T01:09:27.613 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T01:09:27.614 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T01:09:27.616 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T01:09:27.617 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T01:09:27.618 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T01:09:27.618 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T01:09:27.644 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T01:09:27.647 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T01:09:27.811 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T01:09:27.895 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T01:09:27.996 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T01:09:28.030 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T01:09:28.081 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T01:09:31.641 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 13.2 MB in 2s (6,939 kB/s) 2024-04-26T01:09:32.239 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 13.2 MB in 2s (6,847 kB/s) 2024-04-26T01:09:32.260 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 13.2 MB in 2s (5,517 kB/s) 2024-04-26T01:09:32.659 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 13.2 MB in 3s (5,243 kB/s) 2024-04-26T01:09:33.082 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2024-04-26T01:09:33.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:09:33.156 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2024-04-26T01:09:33.356 INFO:teuthology.orchestra.run.smithi181.stdout:Building dependency tree... 2024-04-26T01:09:33.356 INFO:teuthology.orchestra.run.smithi181.stdout:Reading state information... 2024-04-26T01:09:33.522 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:09:33.523 INFO:teuthology.orchestra.run.smithi181.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:09:33.523 INFO:teuthology.orchestra.run.smithi181.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:09:33.523 INFO:teuthology.orchestra.run.smithi181.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:09:33.523 INFO:teuthology.orchestra.run.smithi181.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T01:09:33.524 INFO:teuthology.orchestra.run.smithi181.stdout: python2.7-minimal 2024-04-26T01:09:33.524 INFO:teuthology.orchestra.run.smithi181.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:09:33.524 INFO:teuthology.orchestra.run.smithi181.stdout:The following additional packages will be installed: 2024-04-26T01:09:33.524 INFO:teuthology.orchestra.run.smithi181.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:33.524 INFO:teuthology.orchestra.run.smithi181.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T01:09:33.525 INFO:teuthology.orchestra.run.smithi181.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:33.526 INFO:teuthology.orchestra.run.smithi181.stdout:Suggested packages: 2024-04-26T01:09:33.526 INFO:teuthology.orchestra.run.smithi181.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T01:09:33.526 INFO:teuthology.orchestra.run.smithi181.stdout:Recommended packages: 2024-04-26T01:09:33.526 INFO:teuthology.orchestra.run.smithi181.stdout: thermald 2024-04-26T01:09:33.570 INFO:teuthology.orchestra.run.smithi181.stdout:The following NEW packages will be installed: 2024-04-26T01:09:33.570 INFO:teuthology.orchestra.run.smithi181.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:33.570 INFO:teuthology.orchestra.run.smithi181.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T01:09:33.571 INFO:teuthology.orchestra.run.smithi181.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:33.572 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages will be upgraded: 2024-04-26T01:09:33.572 INFO:teuthology.orchestra.run.smithi181.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T01:09:33.645 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-26T01:09:33.661 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:09:33.669 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T01:09:33.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:09:33.721 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-26T01:09:33.750 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T01:09:33.900 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2024-04-26T01:09:33.901 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2024-04-26T01:09:33.955 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-04-26T01:09:33.955 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-04-26T01:09:34.053 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:09:34.053 INFO:teuthology.orchestra.run.smithi071.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:09:34.053 INFO:teuthology.orchestra.run.smithi071.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:09:34.053 INFO:teuthology.orchestra.run.smithi071.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:09:34.053 INFO:teuthology.orchestra.run.smithi071.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T01:09:34.053 INFO:teuthology.orchestra.run.smithi071.stdout: python2.7-minimal 2024-04-26T01:09:34.054 INFO:teuthology.orchestra.run.smithi071.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:09:34.055 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2024-04-26T01:09:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T01:09:34.055 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:34.056 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2024-04-26T01:09:34.056 INFO:teuthology.orchestra.run.smithi071.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T01:09:34.056 INFO:teuthology.orchestra.run.smithi071.stdout:Recommended packages: 2024-04-26T01:09:34.057 INFO:teuthology.orchestra.run.smithi071.stdout: thermald 2024-04-26T01:09:34.104 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2024-04-26T01:09:34.104 INFO:teuthology.orchestra.run.smithi071.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:34.104 INFO:teuthology.orchestra.run.smithi071.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T01:09:34.104 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T01:09:34.105 INFO:teuthology.orchestra.run.smithi071.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:34.105 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages will be upgraded: 2024-04-26T01:09:34.106 INFO:teuthology.orchestra.run.smithi071.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T01:09:34.123 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:09:34.130 INFO:teuthology.orchestra.run.smithi181.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:09:34.130 INFO:teuthology.orchestra.run.smithi181.stdout:Need to get 113 MB of archives. 2024-04-26T01:09:34.130 INFO:teuthology.orchestra.run.smithi181.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T01:09:34.130 INFO:teuthology.orchestra.run.smithi181.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T01:09:34.138 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:09:34.138 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:09:34.139 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:09:34.139 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:09:34.139 INFO:teuthology.orchestra.run.smithi045.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T01:09:34.139 INFO:teuthology.orchestra.run.smithi045.stdout: python2.7-minimal 2024-04-26T01:09:34.139 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:09:34.141 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2024-04-26T01:09:34.141 INFO:teuthology.orchestra.run.smithi045.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:34.141 INFO:teuthology.orchestra.run.smithi045.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T01:09:34.141 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:34.143 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2024-04-26T01:09:34.143 INFO:teuthology.orchestra.run.smithi045.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T01:09:34.143 INFO:teuthology.orchestra.run.smithi045.stdout:Recommended packages: 2024-04-26T01:09:34.143 INFO:teuthology.orchestra.run.smithi045.stdout: thermald 2024-04-26T01:09:34.181 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T01:09:34.201 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2024-04-26T01:09:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:34.201 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T01:09:34.202 INFO:teuthology.orchestra.run.smithi045.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:34.203 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages will be upgraded: 2024-04-26T01:09:34.204 INFO:teuthology.orchestra.run.smithi045.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T01:09:34.265 INFO:teuthology.orchestra.run.smithi045.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:09:34.266 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 113 MB of archives. 2024-04-26T01:09:34.266 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T01:09:34.266 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T01:09:34.390 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-26T01:09:34.391 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-26T01:09:34.554 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:09:34.554 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:09:34.554 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:09:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:09:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T01:09:34.555 INFO:teuthology.orchestra.run.smithi037.stdout: python2.7-minimal 2024-04-26T01:09:34.555 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:09:34.556 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-04-26T01:09:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T01:09:34.556 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:34.557 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-04-26T01:09:34.557 INFO:teuthology.orchestra.run.smithi037.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T01:09:34.557 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-04-26T01:09:34.557 INFO:teuthology.orchestra.run.smithi037.stdout: thermald 2024-04-26T01:09:34.613 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-04-26T01:09:34.613 INFO:teuthology.orchestra.run.smithi037.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T01:09:34.613 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T01:09:34.614 INFO:teuthology.orchestra.run.smithi037.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T01:09:34.615 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages will be upgraded: 2024-04-26T01:09:34.615 INFO:teuthology.orchestra.run.smithi037.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T01:09:34.656 INFO:teuthology.orchestra.run.smithi181.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T01:09:34.786 INFO:teuthology.orchestra.run.smithi071.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:09:34.786 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 113 MB of archives. 2024-04-26T01:09:34.786 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T01:09:34.786 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T01:09:34.795 INFO:teuthology.orchestra.run.smithi037.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:09:34.795 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 113 MB of archives. 2024-04-26T01:09:34.795 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T01:09:34.795 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T01:09:34.857 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T01:09:34.872 INFO:teuthology.orchestra.run.smithi181.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T01:09:35.045 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T01:09:36.201 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T01:09:36.232 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T01:09:36.556 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T01:09:36.587 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T01:09:37.086 INFO:teuthology.orchestra.run.smithi181.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T01:09:37.087 INFO:teuthology.orchestra.run.smithi181.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T01:09:37.087 INFO:teuthology.orchestra.run.smithi181.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T01:09:37.381 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T01:09:37.382 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T01:09:37.382 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T01:09:37.457 INFO:teuthology.orchestra.run.smithi181.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T01:09:37.511 INFO:teuthology.orchestra.run.smithi181.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T01:09:37.608 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T01:09:37.653 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T01:09:37.890 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 113 MB in 4s (28.8 MB/s) 2024-04-26T01:09:38.055 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T01:09:38.057 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 113 MB in 3s (32.9 MB/s) 2024-04-26T01:09:38.235 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T01:09:38.926 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T01:09:38.926 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T01:09:38.926 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T01:09:38.949 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T01:09:38.949 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T01:09:38.950 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T01:09:39.585 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T01:09:39.698 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T01:09:39.783 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T01:09:39.867 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T01:09:40.082 INFO:teuthology.orchestra.run.smithi181.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T01:09:40.086 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:40.228 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:40.238 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 113 MB in 5s (22.3 MB/s) 2024-04-26T01:09:40.255 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 113 MB in 6s (19.7 MB/s) 2024-04-26T01:09:40.328 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T01:09:40.332 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:40.449 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T01:09:40.466 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:40.526 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T01:09:42.522 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T01:09:42.527 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:42.631 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:42.678 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T01:09:42.682 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:42.712 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T01:09:42.733 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:42.802 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:42.904 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:42.969 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T01:09:42.991 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:43.068 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:43.290 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T01:09:43.311 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:43.348 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:43.664 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T01:09:43.684 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:43.713 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:44.773 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T01:09:44.794 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:44.880 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:44.906 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T01:09:44.927 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:45.021 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:45.376 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T01:09:45.397 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:45.434 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:45.442 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T01:09:45.462 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:45.500 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:49.799 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:49.917 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:50.258 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:50.353 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:50.396 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:50.494 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:50.698 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T01:09:50.710 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T01:09:50.748 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:09:50.819 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:50.964 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:51.259 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T01:09:51.282 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T01:09:51.317 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:09:51.623 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:51.750 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:52.066 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:52.162 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:52.176 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:52.327 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:52.440 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T01:09:52.450 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T01:09:52.543 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:09:52.683 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:52.789 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:09:53.058 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T01:09:53.081 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T01:09:53.125 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:09:56.260 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T01:09:56.297 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:56.335 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:56.881 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T01:09:56.918 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:56.955 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:58.637 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T01:09:58.676 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:58.720 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:59.574 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:09:59.592 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T01:09:59.630 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T01:09:59.658 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:09:59.679 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:10:00.029 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:10:00.151 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:00.272 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:10:00.285 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T01:10:00.394 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:10:00.414 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:00.746 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:10:00.883 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:01.735 INFO:teuthology.orchestra.run.smithi181.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:01.735 INFO:teuthology.orchestra.run.smithi181.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:01.736 INFO:teuthology.orchestra.run.smithi181.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:01.736 INFO:teuthology.orchestra.run.smithi181.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:01.776 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T01:10:01.891 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:01.900 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:02.133 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:10:02.255 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:10:02.605 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:10:02.727 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:02.853 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T01:10:02.989 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:03.270 INFO:teuthology.orchestra.run.smithi045.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:03.270 INFO:teuthology.orchestra.run.smithi045.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:03.270 INFO:teuthology.orchestra.run.smithi045.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:03.270 INFO:teuthology.orchestra.run.smithi045.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:03.277 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:03.311 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T01:10:03.424 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:03.435 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T01:10:03.810 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T01:10:03.956 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:04.082 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T01:10:04.232 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:04.311 INFO:teuthology.orchestra.run.smithi071.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:04.312 INFO:teuthology.orchestra.run.smithi071.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:04.312 INFO:teuthology.orchestra.run.smithi071.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:04.312 INFO:teuthology.orchestra.run.smithi071.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:04.482 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:04.661 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T01:10:04.786 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T01:10:04.859 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:04.913 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:04.977 INFO:teuthology.orchestra.run.smithi181.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T01:10:04.977 INFO:teuthology.orchestra.run.smithi181.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:05.631 INFO:teuthology.orchestra.run.smithi037.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:05.631 INFO:teuthology.orchestra.run.smithi037.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:05.631 INFO:teuthology.orchestra.run.smithi037.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:05.631 INFO:teuthology.orchestra.run.smithi037.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:05.789 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:05.884 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:06.252 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T01:10:06.386 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T01:10:06.512 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:06.574 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T01:10:06.575 INFO:teuthology.orchestra.run.smithi045.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:07.669 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T01:10:07.786 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T01:10:07.876 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:07.912 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:07.979 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T01:10:07.980 INFO:teuthology.orchestra.run.smithi071.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:09.343 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T01:10:09.477 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T01:10:09.628 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T01:10:09.684 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T01:10:09.684 INFO:teuthology.orchestra.run.smithi037.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:19.474 INFO:teuthology.orchestra.run.smithi181.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T01:10:19.474 INFO:teuthology.orchestra.run.smithi181.stdout:Sourcing file `/etc/default/grub' 2024-04-26T01:10:19.502 INFO:teuthology.orchestra.run.smithi181.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:19.510 INFO:teuthology.orchestra.run.smithi181.stdout:Generating grub configuration file ... 2024-04-26T01:10:19.827 INFO:teuthology.orchestra.run.smithi181.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:19.846 INFO:teuthology.orchestra.run.smithi181.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:20.135 INFO:teuthology.orchestra.run.smithi181.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:20.140 INFO:teuthology.orchestra.run.smithi181.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:20.199 INFO:teuthology.orchestra.run.smithi181.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:20.204 INFO:teuthology.orchestra.run.smithi181.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:20.348 INFO:teuthology.orchestra.run.smithi181.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:20.348 INFO:teuthology.orchestra.run.smithi181.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:20.349 INFO:teuthology.orchestra.run.smithi181.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:20.385 INFO:teuthology.orchestra.run.smithi181.stdout:done 2024-04-26T01:10:20.858 DEBUG:teuthology.orchestra.run.smithi181:> dpkg -s linux-image-generic 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Package: linux-image-generic 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Status: install ok installed 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Priority: optional 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Section: kernel 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Installed-Size: 21 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: amd64 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Source: linux-meta 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:20.876 INFO:teuthology.orchestra.run.smithi181.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:20.877 INFO:teuthology.orchestra.run.smithi181.stdout:Recommends: thermald 2024-04-26T01:10:20.877 INFO:teuthology.orchestra.run.smithi181.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:20.877 INFO:teuthology.orchestra.run.smithi181.stdout: available. 2024-04-26T01:10:20.877 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T01:10:20.877 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T01:10:20.877 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T01:10:20.877 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-04-26T01:10:20.878 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T01:10:20.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get clean 2024-04-26T01:10:21.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get update 2024-04-26T01:10:21.327 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:10:21.349 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:10:21.398 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:10:21.480 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:10:21.640 INFO:teuthology.orchestra.run.smithi045.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T01:10:21.640 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub' 2024-04-26T01:10:21.670 INFO:teuthology.orchestra.run.smithi045.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:21.689 INFO:teuthology.orchestra.run.smithi045.stdout:Generating grub configuration file ... 2024-04-26T01:10:21.833 INFO:teuthology.orchestra.run.smithi071.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T01:10:21.833 INFO:teuthology.orchestra.run.smithi071.stdout:Sourcing file `/etc/default/grub' 2024-04-26T01:10:21.844 INFO:teuthology.orchestra.run.smithi071.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:21.862 INFO:teuthology.orchestra.run.smithi071.stdout:Generating grub configuration file ... 2024-04-26T01:10:21.953 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:21.977 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:22.119 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:22.140 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:22.287 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:22.294 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:22.363 INFO:teuthology.orchestra.run.smithi045.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:22.369 INFO:teuthology.orchestra.run.smithi045.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:22.418 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:22.423 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:22.493 INFO:teuthology.orchestra.run.smithi071.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:22.498 INFO:teuthology.orchestra.run.smithi071.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:22.527 INFO:teuthology.orchestra.run.smithi045.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:22.527 INFO:teuthology.orchestra.run.smithi045.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:22.527 INFO:teuthology.orchestra.run.smithi045.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:22.575 INFO:teuthology.orchestra.run.smithi045.stdout:done 2024-04-26T01:10:22.662 INFO:teuthology.orchestra.run.smithi071.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:22.662 INFO:teuthology.orchestra.run.smithi071.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:22.662 INFO:teuthology.orchestra.run.smithi071.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:22.665 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2024-04-26T01:10:22.683 DEBUG:teuthology.orchestra.run.smithi181:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:10:22.698 INFO:teuthology.orchestra.run.smithi071.stdout:done 2024-04-26T01:10:22.736 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2024-04-26T01:10:22.913 INFO:teuthology.orchestra.run.smithi181.stdout:Building dependency tree... 2024-04-26T01:10:22.914 INFO:teuthology.orchestra.run.smithi181.stdout:Reading state information... 2024-04-26T01:10:23.052 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic 2024-04-26T01:10:23.067 INFO:teuthology.orchestra.run.smithi181.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T01:10:23.068 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:10:23.068 INFO:teuthology.orchestra.run.smithi181.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:10:23.068 INFO:teuthology.orchestra.run.smithi181.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:10:23.068 INFO:teuthology.orchestra.run.smithi181.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:10:23.068 INFO:teuthology.orchestra.run.smithi181.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:10:23.068 INFO:teuthology.orchestra.run.smithi181.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:10:23.069 INFO:teuthology.orchestra.run.smithi181.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 21 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:23.078 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2024-04-26T01:10:23.079 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:23.079 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:23.079 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2024-04-26T01:10:23.079 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T01:10:23.079 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T01:10:23.079 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T01:10:23.079 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-26T01:10:23.079 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T01:10:23.079 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get clean 2024-04-26T01:10:23.107 INFO:teuthology.orchestra.run.smithi181.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:10:23.109 DEBUG:teuthology.orchestra.run.smithi181:> dpkg -s linux-image-generic 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Package: linux-image-generic 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Status: install ok installed 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Priority: optional 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Section: kernel 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Installed-Size: 21 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: amd64 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Source: linux-meta 2024-04-26T01:10:23.167 INFO:teuthology.orchestra.run.smithi181.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:23.168 INFO:teuthology.orchestra.run.smithi181.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:23.168 INFO:teuthology.orchestra.run.smithi181.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:23.168 INFO:teuthology.orchestra.run.smithi181.stdout:Recommends: thermald 2024-04-26T01:10:23.168 INFO:teuthology.orchestra.run.smithi181.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:23.168 INFO:teuthology.orchestra.run.smithi181.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:23.168 INFO:teuthology.orchestra.run.smithi181.stdout: available. 2024-04-26T01:10:23.168 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-04-26T01:10:23.170 DEBUG:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic 2024-04-26T01:10:23.190 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic 2024-04-26T01:10:23.190 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 21 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:23.191 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2024-04-26T01:10:23.192 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T01:10:23.192 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T01:10:23.192 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T01:10:23.192 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-04-26T01:10:23.192 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T01:10:23.192 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get clean 2024-04-26T01:10:23.216 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.Tn6NvMDCIj 2024-04-26T01:10:23.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Tn6NvMDCIj 2024-04-26T01:10:23.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.Tn6NvMDCIj 2024-04-26T01:10:23.392 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2024-04-26T01:10:23.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-04-26T01:10:23.396 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.Tn6NvMDCIj is 10KB 2024-04-26T01:10:23.411 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.Tn6NvMDCIj 2024-04-26T01:10:23.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:23.473 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:10:23.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T01:10:23.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:23.519 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:10:23.519 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:10:23.531 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:10:23.534 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:10:23.534 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:10:23.535 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:10:23.547 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T01:10:23.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo update-grub 2024-04-26T01:10:23.554 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:10:23.558 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:10:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T01:10:24.584 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub' 2024-04-26T01:10:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:24.633 INFO:teuthology.orchestra.run.smithi037.stdout:Generating grub configuration file ... 2024-04-26T01:10:24.767 INFO:teuthology.orchestra.run.smithi181.stderr:Sourcing file `/etc/default/grub' 2024-04-26T01:10:24.769 INFO:teuthology.orchestra.run.smithi181.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:24.781 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-04-26T01:10:24.902 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-26T01:10:24.913 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:24.919 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:10:24.937 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T01:10:24.941 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:24.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:10:24.969 INFO:teuthology.orchestra.run.smithi181.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:24.975 INFO:teuthology.orchestra.run.smithi181.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:24.978 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-26T01:10:25.018 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T01:10:25.178 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2024-04-26T01:10:25.178 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2024-04-26T01:10:25.211 INFO:teuthology.orchestra.run.smithi181.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:25.215 INFO:teuthology.orchestra.run.smithi181.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:25.217 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-04-26T01:10:25.218 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-04-26T01:10:25.253 INFO:teuthology.orchestra.run.smithi181.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:25.258 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:25.258 INFO:teuthology.orchestra.run.smithi181.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:25.260 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:25.326 INFO:teuthology.orchestra.run.smithi037.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:25.331 INFO:teuthology.orchestra.run.smithi037.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:25.354 INFO:teuthology.orchestra.run.smithi071.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T01:10:25.354 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:10:25.354 INFO:teuthology.orchestra.run.smithi071.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:10:25.354 INFO:teuthology.orchestra.run.smithi071.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:10:25.354 INFO:teuthology.orchestra.run.smithi071.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:10:25.355 INFO:teuthology.orchestra.run.smithi071.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:10:25.355 INFO:teuthology.orchestra.run.smithi071.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:10:25.355 INFO:teuthology.orchestra.run.smithi071.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:10:25.394 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:10:25.396 DEBUG:teuthology.orchestra.run.smithi071:> dpkg -s linux-image-generic 2024-04-26T01:10:25.402 INFO:teuthology.orchestra.run.smithi045.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T01:10:25.402 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:10:25.402 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:10:25.402 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:10:25.402 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:10:25.403 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:10:25.403 INFO:teuthology.orchestra.run.smithi045.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:10:25.403 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:10:25.410 INFO:teuthology.orchestra.run.smithi181.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:25.410 INFO:teuthology.orchestra.run.smithi181.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:25.410 INFO:teuthology.orchestra.run.smithi181.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:25.411 INFO:teuthology.orchestra.run.smithi071.stdout:Package: linux-image-generic 2024-04-26T01:10:25.411 INFO:teuthology.orchestra.run.smithi071.stdout:Status: install ok installed 2024-04-26T01:10:25.411 INFO:teuthology.orchestra.run.smithi071.stdout:Priority: optional 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Section: kernel 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Installed-Size: 21 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: amd64 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Source: linux-meta 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Recommends: thermald 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:25.412 INFO:teuthology.orchestra.run.smithi071.stdout: available. 2024-04-26T01:10:25.413 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-04-26T01:10:25.431 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-04-26T01:10:25.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-04-26T01:10:25.449 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:10:25.451 DEBUG:teuthology.orchestra.run.smithi045:> dpkg -s linux-image-generic 2024-04-26T01:10:25.459 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.VmJSvuCqpz 2024-04-26T01:10:25.459 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub/grub.cfg /tmp/tmp.VmJSvuCqpz 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Package: linux-image-generic 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Status: install ok installed 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Priority: optional 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Section: kernel 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Installed-Size: 21 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: amd64 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Source: linux-meta 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:25.467 INFO:teuthology.orchestra.run.smithi045.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:25.468 INFO:teuthology.orchestra.run.smithi045.stdout:Recommends: thermald 2024-04-26T01:10:25.468 INFO:teuthology.orchestra.run.smithi045.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:25.468 INFO:teuthology.orchestra.run.smithi045.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:25.468 INFO:teuthology.orchestra.run.smithi045.stdout: available. 2024-04-26T01:10:25.468 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-26T01:10:25.514 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.reYVG4JtxM 2024-04-26T01:10:25.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub/grub.cfg /tmp/tmp.reYVG4JtxM 2024-04-26T01:10:25.516 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.VmJSvuCqpz 2024-04-26T01:10:25.516 INFO:teuthology.orchestra.run.smithi037.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:25.516 INFO:teuthology.orchestra.run.smithi037.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:25.516 INFO:teuthology.orchestra.run.smithi037.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:25.561 INFO:teuthology.orchestra.run.smithi037.stdout:done 2024-04-26T01:10:25.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.reYVG4JtxM 2024-04-26T01:10:25.645 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.VmJSvuCqpz is 10KB 2024-04-26T01:10:25.658 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.VmJSvuCqpz 2024-04-26T01:10:25.663 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:25.688 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.reYVG4JtxM is 10KB 2024-04-26T01:10:25.701 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.reYVG4JtxM 2024-04-26T01:10:25.706 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:25.719 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:10:25.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T01:10:25.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:25.763 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:10:25.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T01:10:25.763 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:25.787 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T01:10:25.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo update-grub 2024-04-26T01:10:25.857 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T01:10:25.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo update-grub 2024-04-26T01:10:26.041 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 21 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-04-26T01:10:26.062 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-04-26T01:10:26.063 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:26.063 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:26.063 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:26.063 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-04-26T01:10:26.063 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:26.063 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:26.063 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-04-26T01:10:26.063 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T01:10:26.063 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T01:10:26.063 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T01:10:26.063 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T01:10:26.063 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T01:10:26.064 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get clean 2024-04-26T01:10:26.226 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-04-26T01:10:26.344 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:10:26.344 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:10:26.357 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:10:26.378 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:10:26.934 INFO:teuthology.orchestra.run.smithi071.stderr:Sourcing file `/etc/default/grub' 2024-04-26T01:10:26.935 INFO:teuthology.orchestra.run.smithi071.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:26.938 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-04-26T01:10:27.123 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:27.130 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:27.263 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub' 2024-04-26T01:10:27.265 INFO:teuthology.orchestra.run.smithi045.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:27.267 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-04-26T01:10:27.413 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:27.418 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:27.460 INFO:teuthology.orchestra.run.smithi071.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:27.464 INFO:teuthology.orchestra.run.smithi071.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:27.496 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:27.503 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:27.608 INFO:teuthology.orchestra.run.smithi071.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:27.608 INFO:teuthology.orchestra.run.smithi071.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:27.608 INFO:teuthology.orchestra.run.smithi071.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:27.628 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-04-26T01:10:27.645 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-04-26T01:10:27.717 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:27.723 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:27.766 INFO:teuthology.orchestra.run.smithi045.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:27.767 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T01:10:27.770 INFO:teuthology.orchestra.run.smithi045.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:27.785 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T01:10:27.838 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T01:10:27.913 INFO:teuthology.orchestra.run.smithi045.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:27.913 INFO:teuthology.orchestra.run.smithi045.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:27.913 INFO:teuthology.orchestra.run.smithi045.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:27.939 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-04-26T01:10:27.941 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-04-26T01:10:28.022 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-26T01:10:28.022 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-26T01:10:28.187 INFO:teuthology.orchestra.run.smithi037.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T01:10:28.187 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:10:28.187 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:10:28.187 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:10:28.187 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:10:28.187 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:10:28.188 INFO:teuthology.orchestra.run.smithi037.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:10:28.188 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:10:28.228 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T01:10:28.230 DEBUG:teuthology.orchestra.run.smithi037:> dpkg -s linux-image-generic 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Package: linux-image-generic 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Status: install ok installed 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Priority: optional 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Section: kernel 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Installed-Size: 21 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: amd64 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Source: linux-meta 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Version: 5.15.0.105.102 2024-04-26T01:10:28.290 INFO:teuthology.orchestra.run.smithi037.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T01:10:28.291 INFO:teuthology.orchestra.run.smithi037.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T01:10:28.291 INFO:teuthology.orchestra.run.smithi037.stdout:Recommends: thermald 2024-04-26T01:10:28.291 INFO:teuthology.orchestra.run.smithi037.stdout:Description: Generic Linux kernel image 2024-04-26T01:10:28.291 INFO:teuthology.orchestra.run.smithi037.stdout: This package will always depend on the latest generic kernel image 2024-04-26T01:10:28.291 INFO:teuthology.orchestra.run.smithi037.stdout: available. 2024-04-26T01:10:28.291 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-04-26T01:10:28.338 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.A4Mtkfdcoa 2024-04-26T01:10:28.339 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub/grub.cfg /tmp/tmp.A4Mtkfdcoa 2024-04-26T01:10:28.395 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.A4Mtkfdcoa 2024-04-26T01:10:28.513 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.A4Mtkfdcoa is 10KB 2024-04-26T01:10:28.526 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.A4Mtkfdcoa 2024-04-26T01:10:28.531 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:28.591 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:10:28.591 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T01:10:28.591 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T01:10:28.682 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T01:10:28.682 DEBUG:teuthology.orchestra.run.smithi037:> sudo update-grub 2024-04-26T01:10:30.069 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub' 2024-04-26T01:10:30.071 INFO:teuthology.orchestra.run.smithi037.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T01:10:30.074 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-04-26T01:10:30.271 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T01:10:30.276 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T01:10:30.523 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T01:10:30.528 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T01:10:30.566 INFO:teuthology.orchestra.run.smithi037.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T01:10:30.571 INFO:teuthology.orchestra.run.smithi037.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T01:10:30.734 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T01:10:30.734 INFO:teuthology.orchestra.run.smithi037.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T01:10:30.735 INFO:teuthology.orchestra.run.smithi037.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T01:10:30.752 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-04-26T01:10:30.754 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-04-26T01:10:55.466 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-04-26T01:10:55.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-26T01:10:55.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:10:57.650 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-04-26T01:10:57.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-26T01:10:57.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:10:57.944 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-26T01:10:57.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T01:10:57.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:00.758 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T01:11:00.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T01:11:00.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:13.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-04-26T01:11:15.998 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-04-26T01:11:16.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-04-26T01:11:19.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-26T01:11:22.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-26T01:11:22.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:25.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-26T01:11:25.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:25.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T01:11:25.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:26.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-04-26T01:11:28.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-04-26T01:11:28.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T01:11:28.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:38.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-26T01:11:38.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:40.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-26T01:11:40.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:46.718 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-26T01:11:58.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T01:11:58.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:11:58.733 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-04-26T01:12:09.800 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-04-26T01:12:10.398 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-26T01:12:10.398 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-105-generic"... 2024-04-26T01:12:10.402 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-04-26T01:12:10.447 INFO:teuthology.orchestra.run.smithi181.stdout:5.15.0-105-generic 2024-04-26T01:12:10.447 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T01:12:10.447 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T01:12:10.447 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T01:12:11.448 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-04-26T01:12:11.449 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T01:12:11.521 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-04-26T01:12:11.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-04-26T01:12:11.552 DEBUG:teuthology.parallel:result is None 2024-04-26T01:12:13.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T01:12:13.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:12:13.985 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-04-26T01:12:14.579 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T01:12:14.579 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-26T01:12:14.579 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-04-26T01:12:14.628 INFO:teuthology.orchestra.run.smithi037.stdout:5.15.0-105-generic 2024-04-26T01:12:14.628 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T01:12:14.629 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T01:12:14.629 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T01:12:15.630 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T01:12:15.630 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T01:12:15.702 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-04-26T01:12:15.729 DEBUG:teuthology.parallel:result is None 2024-04-26T01:12:25.515 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T01:12:26.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-26T01:12:26.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:12:26.935 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-04-26T01:12:27.475 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-26T01:12:27.475 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-105-generic"... 2024-04-26T01:12:27.475 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-04-26T01:12:27.522 INFO:teuthology.orchestra.run.smithi071.stdout:5.15.0-105-generic 2024-04-26T01:12:27.522 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T01:12:27.523 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T01:12:27.523 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T01:12:28.523 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-26T01:12:28.523 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T01:12:28.594 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-04-26T01:12:28.616 DEBUG:teuthology.parallel:result is None 2024-04-26T01:12:37.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T01:12:37.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:12:37.769 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-26T01:12:38.313 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T01:12:38.313 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-26T01:12:38.313 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-04-26T01:12:38.359 INFO:teuthology.orchestra.run.smithi045.stdout:5.15.0-105-generic 2024-04-26T01:12:38.360 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T01:12:38.360 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T01:12:38.360 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T01:12:39.361 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-26T01:12:39.361 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T01:12:39.436 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-04-26T01:12:39.457 DEBUG:teuthology.parallel:result is None 2024-04-26T01:12:39.457 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T01:12:39.464 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T01:12:39.464 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:12:39.466 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:12:39.469 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:12:39.472 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T01:12:39.478 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T01:12:39.485 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T01:12:39.492 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T01:12:39.493 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:12:39.513 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:12:39.517 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:12:39.520 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T01:12:39.549 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T01:12:39.555 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T01:12:39.556 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:12:39.565 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:12:39.572 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:12:39.579 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T01:12:39.584 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.593 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.593 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.596 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.601 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.604 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.606 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.614 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T01:12:39.616 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T01:12:39.622 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T01:12:39.623 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:12:39.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:12:39.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:12:39.651 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T01:12:39.671 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T01:12:39.680 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T01:12:39.680 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:12:39.693 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:12:39.705 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:12:39.707 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T01:12:39.718 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:12:39.744 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:12:39.792 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:12:39.792 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:12:39.852 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:12:39.858 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:12:39.904 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:12:39.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:12:39.977 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:12:39.983 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:12:40.030 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:12:40.031 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:12:40.106 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T01:12:40.112 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T01:12:40.158 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:12:40.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T01:12:40.225 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-04-26T01:12:40.227 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-04-26T01:12:40.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-04-26T01:12:40.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-04-26T01:12:40.305 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T01:12:40.314 INFO:teuthology.task.internal:Starting timer... 2024-04-26T01:12:40.314 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T01:12:40.381 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T01:12:40.390 DEBUG:teuthology.task.selinux:Excluding smithi037: OS 'ubuntu' does not support SELinux 2024-04-26T01:12:40.390 DEBUG:teuthology.task.selinux:Excluding smithi045: OS 'ubuntu' does not support SELinux 2024-04-26T01:12:40.390 DEBUG:teuthology.task.selinux:Excluding smithi071: OS 'ubuntu' does not support SELinux 2024-04-26T01:12:40.390 DEBUG:teuthology.task.selinux:Excluding smithi181: OS 'ubuntu' does not support SELinux 2024-04-26T01:12:40.391 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T01:12:40.391 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-26T01:12:40.391 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T01:12:40.391 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T01:12:40.400 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T01:12:40.406 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T01:12:40.531 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T01:12:40.545 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T01:12:40.546 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi045.front.sepia.ceph.com,smithi071.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T01:19:53.383 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-04-26T01:19:53.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T01:19:53.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:19:53.475 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-04-26T01:19:53.554 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-26T01:19:53.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T01:19:53.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:19:53.647 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-04-26T01:19:53.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-04-26T01:19:53.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-26T01:19:53.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:19:53.814 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-04-26T01:19:53.893 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-26T01:19:53.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-26T01:19:53.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T01:19:53.974 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-04-26T01:19:54.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-04-26T01:19:54.051 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T01:19:54.062 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T01:19:54.062 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:19:54.062 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:19:54.065 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:19:54.065 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:19:54.067 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:19:54.068 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:19:54.070 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T01:19:54.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:19:54.094 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:19:54.094 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Command line: ntpd -gq 2024-04-26T01:19:54.094 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: ---------------------------------------------------- 2024-04-26T01:19:54.095 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:19:54.095 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:19:54.095 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: corporation. Support and training for ntp-4 are 2024-04-26T01:19:54.095 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: available at https://www.nwtime.org/support 2024-04-26T01:19:54.095 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: ---------------------------------------------------- 2024-04-26T01:19:54.095 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: proto: precision = 0.065 usec (-24) 2024-04-26T01:19:54.096 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: basedate set to 2022-02-04 2024-04-26T01:19:54.096 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:19:54.096 INFO:teuthology.orchestra.run.smithi045.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.096 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.096 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.097 INFO:teuthology.orchestra.run.smithi045.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.097 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:19:54.097 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:19:54.098 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:19:54.098 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Listen normally on 3 ens1f0 172.21.15.45:123 2024-04-26T01:19:54.098 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Listen normally on 4 lo [::1]:123 2024-04-26T01:19:54.098 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12be%4]:123 2024-04-26T01:19:54.098 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:19:54 ntpd[17114]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Command line: ntpd -gq 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: ---------------------------------------------------- 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: corporation. Support and training for ntp-4 are 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: available at https://www.nwtime.org/support 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: ---------------------------------------------------- 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: proto: precision = 0.046 usec (-24) 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: basedate set to 2022-02-04 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:19:54.099 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.100 INFO:teuthology.orchestra.run.smithi037.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.100 INFO:teuthology.orchestra.run.smithi037.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.101 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.101 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:19:54.102 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:19:54.102 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:19:54.102 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Listen normally on 3 ens1f0 172.21.15.37:123 2024-04-26T01:19:54.102 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Listen normally on 4 lo [::1]:123 2024-04-26T01:19:54.102 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe88:7794%4]:123 2024-04-26T01:19:54.102 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:19:54 ntpd[17171]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Command line: ntpd -gq 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: ---------------------------------------------------- 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: corporation. Support and training for ntp-4 are 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: available at https://www.nwtime.org/support 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: ---------------------------------------------------- 2024-04-26T01:19:54.103 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: proto: precision = 0.060 usec (-24) 2024-04-26T01:19:54.104 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: basedate set to 2022-02-04 2024-04-26T01:19:54.104 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:19:54.104 INFO:teuthology.orchestra.run.smithi071.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.104 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.104 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.105 INFO:teuthology.orchestra.run.smithi071.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.105 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:19:54.106 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:19:54.106 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:19:54.106 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Listen normally on 3 enp3s0f1 172.21.15.71:123 2024-04-26T01:19:54.106 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Listen normally on 4 lo [::1]:123 2024-04-26T01:19:54.106 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d765%5]:123 2024-04-26T01:19:54.106 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:19:54 ntpd[17241]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:19:54.113 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T01:19:54.113 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Command line: ntpd -gq 2024-04-26T01:19:54.113 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: ---------------------------------------------------- 2024-04-26T01:19:54.113 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T01:19:54.113 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T01:19:54.113 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: corporation. Support and training for ntp-4 are 2024-04-26T01:19:54.113 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: available at https://www.nwtime.org/support 2024-04-26T01:19:54.114 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: ---------------------------------------------------- 2024-04-26T01:19:54.114 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: proto: precision = 0.045 usec (-24) 2024-04-26T01:19:54.114 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: basedate set to 2022-02-04 2024-04-26T01:19:54.114 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: gps base set to 2022-02-06 (week 2196) 2024-04-26T01:19:54.115 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.115 INFO:teuthology.orchestra.run.smithi181.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T01:19:54.115 INFO:teuthology.orchestra.run.smithi181.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Listen normally on 3 enp3s0f1 172.21.15.181:123 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Listen normally on 4 lo [::1]:123 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7b9d%5]:123 2024-04-26T01:19:54.116 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:19:54 ntpd[17205]: Listening on routing socket on fd #22 for interface updates 2024-04-26T01:19:55.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:55.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:55 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:55 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:55 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:55 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:55 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:55 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:55.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:55 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:55.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:55 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:56.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:56 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:56 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:56.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:56 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:56.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:56 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:56 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:56 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:56.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:56 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:56.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:56 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:56.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:56 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:57.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:57 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:57.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:57 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:57 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:57 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:57 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:57 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:57.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:57 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:57.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:57.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:57.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:57.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:57 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:57.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:57 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:57 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:57 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:57.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:57 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:57.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:57 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:57.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:57 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:58 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:58 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:58 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:58 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:58 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:58.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:58 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:58.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:58.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:58 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:58.100 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:58 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.100 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:58 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:58 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:58 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:58 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:58.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:58 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:58.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:58.116 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:58 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:59.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:59.098 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:19:59 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:59 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:59 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:59 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.099 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:59 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:59 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:59.100 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:19:59 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:59.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.103 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.103 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:59.103 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:59.103 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.103 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.103 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:59.103 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:19:59 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:19:59.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:59 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:59 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:59 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:19:59.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:59 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:19:59.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:59 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:19:59.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:19:59 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:00.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:00 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:00 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:00 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:00 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:00 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:00.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:00 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:00.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:00.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:00 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:00.100 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:00 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.100 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:00 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:00 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:00 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:00 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:00.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:00 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:00.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:00.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:00.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:00.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:00.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:00 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.095 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:01.096 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:01.096 INFO:teuthology.orchestra.run.smithi045.stdout:26 Apr 01:20:01 ntpd[17114]: ntpd: time slew -0.000196 s 2024-04-26T01:20:01.096 INFO:teuthology.orchestra.run.smithi045.stdout:ntpd: time slew -0.000196s 2024-04-26T01:20:01.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:20:01.097 INFO:teuthology.orchestra.run.smithi045.stderr:26 Apr 01:20:01 ntpd[17114]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:20:01.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.097 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:01.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:01.098 INFO:teuthology.orchestra.run.smithi037.stdout:26 Apr 01:20:01 ntpd[17171]: ntpd: time slew -0.002339 s 2024-04-26T01:20:01.098 INFO:teuthology.orchestra.run.smithi037.stdout:ntpd: time slew -0.002339s 2024-04-26T01:20:01.098 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:20:01.099 INFO:teuthology.orchestra.run.smithi037.stderr:26 Apr 01:20:01 ntpd[17171]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stdout:26 Apr 01:20:01 ntpd[17241]: ntpd: time slew -0.001328 s 2024-04-26T01:20:01.101 INFO:teuthology.orchestra.run.smithi071.stdout:ntpd: time slew -0.001328s 2024-04-26T01:20:01.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:20:01.102 INFO:teuthology.orchestra.run.smithi071.stderr:26 Apr 01:20:01 ntpd[17241]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:20:01.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:01 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:01 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:01 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:01.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:01 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:01.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:01 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:01.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:01 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:20:01.145 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================== 2024-04-26T01:20:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.145 INFO:teuthology.orchestra.run.smithi045.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.146 INFO:teuthology.orchestra.run.smithi045.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.146 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:20:01.147 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-04-26T01:20:01.147 INFO:teuthology.orchestra.run.smithi037.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.147 INFO:teuthology.orchestra.run.smithi037.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.147 INFO:teuthology.orchestra.run.smithi037.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.147 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.153 INFO:teuthology.orchestra.run.smithi071.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:20:01.153 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================== 2024-04-26T01:20:01.154 INFO:teuthology.orchestra.run.smithi071.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.154 INFO:teuthology.orchestra.run.smithi071.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.154 INFO:teuthology.orchestra.run.smithi071.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:01.154 INFO:teuthology.orchestra.run.smithi071.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T01:20:02.114 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T01:20:02.115 INFO:teuthology.orchestra.run.smithi181.stdout:26 Apr 01:20:02 ntpd[17205]: ntpd: time slew -0.000846 s 2024-04-26T01:20:02.115 INFO:teuthology.orchestra.run.smithi181.stdout:ntpd: time slew -0.000846s 2024-04-26T01:20:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T01:20:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:26 Apr 01:20:02 ntpd[17205]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T01:20:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:20:02.171 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================== 2024-04-26T01:20:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:20:02.172 INFO:teuthology.run_tasks:Running task install... 2024-04-26T01:20:02.182 DEBUG:teuthology.task.install:project ceph 2024-04-26T01:20:02.182 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-26T01:20:02.182 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-26T01:20:02.183 INFO:teuthology.task.install:Using flavor: default 2024-04-26T01:20:02.188 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T01:20:02.188 INFO:teuthology.task.install:extra packages: [] 2024-04-26T01:20:02.189 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-key list | grep Ceph 2024-04-26T01:20:02.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-key list | grep Ceph 2024-04-26T01:20:02.190 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-key list | grep Ceph 2024-04-26T01:20:02.190 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-key list | grep Ceph 2024-04-26T01:20:02.252 INFO:teuthology.orchestra.run.smithi071.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:20:02.257 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:20:02.260 INFO:teuthology.orchestra.run.smithi045.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:20:02.269 INFO:teuthology.orchestra.run.smithi181.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T01:20:02.281 INFO:teuthology.orchestra.run.smithi071.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:20:02.281 INFO:teuthology.orchestra.run.smithi071.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:20:02.282 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:20:02.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:20:02.288 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:20:02.288 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:20:02.289 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:20:02.290 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:20:02.294 INFO:teuthology.orchestra.run.smithi045.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:20:02.294 INFO:teuthology.orchestra.run.smithi045.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:20:02.295 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:20:02.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:20:02.301 INFO:teuthology.orchestra.run.smithi181.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T01:20:02.302 INFO:teuthology.orchestra.run.smithi181.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T01:20:02.302 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T01:20:02.302 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:20:02.445 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:20:02.445 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:02.449 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:20:02.449 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:02.468 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:20:02.468 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:02.540 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-26T01:20:02.540 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:02.561 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:20:02.561 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:20:02.575 DEBUG:teuthology.orchestra.run.smithi071:> sudo apt-get update 2024-04-26T01:20:02.620 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:20:02.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:20:02.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo apt-get update 2024-04-26T01:20:02.638 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:20:02.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:20:02.652 DEBUG:teuthology.orchestra.run.smithi181:> sudo apt-get update 2024-04-26T01:20:02.717 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:20:02.717 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T01:20:02.732 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-04-26T01:20:02.774 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:20:02.796 INFO:teuthology.orchestra.run.smithi071.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:20:02.817 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:20:02.839 INFO:teuthology.orchestra.run.smithi071.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:20:02.859 INFO:teuthology.orchestra.run.smithi045.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:20:02.860 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:20:02.878 INFO:teuthology.orchestra.run.smithi181.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:20:02.880 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:20:02.887 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:20:02.899 INFO:teuthology.orchestra.run.smithi181.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:20:02.904 INFO:teuthology.orchestra.run.smithi045.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:20:02.909 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:20:02.909 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:20:02.920 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:20:02.922 INFO:teuthology.orchestra.run.smithi181.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:20:02.925 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:20:02.941 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:20:02.944 INFO:teuthology.orchestra.run.smithi181.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:20:02.956 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:20:02.978 INFO:teuthology.orchestra.run.smithi181.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:20:03.019 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T01:20:03.056 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T01:20:03.092 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T01:20:03.129 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,416 B] 2024-04-26T01:20:03.198 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:20:03.198 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:20:03.269 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:20:03.323 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:7 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:20:03.341 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T01:20:03.341 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T01:20:03.351 INFO:teuthology.orchestra.run.smithi181.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:20:03.406 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:20:03.412 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T01:20:03.443 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T01:20:03.490 INFO:teuthology.orchestra.run.smithi071.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:20:03.493 INFO:teuthology.orchestra.run.smithi045.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T01:20:03.547 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 28.9 kB in 1s (47.6 kB/s) 2024-04-26T01:20:03.573 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 28.9 kB in 1s (39.4 kB/s) 2024-04-26T01:20:03.636 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 28.9 kB in 1s (33.0 kB/s) 2024-04-26T01:20:03.762 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 28.9 kB in 1s (31.1 kB/s) 2024-04-26T01:20:04.549 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T01:20:04.564 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:04.585 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2024-04-26T01:20:04.599 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-26T01:20:04.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:04.615 DEBUG:teuthology.orchestra.run.smithi071:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:04.632 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-26T01:20:04.662 INFO:teuthology.orchestra.run.smithi181.stdout:Reading package lists... 2024-04-26T01:20:04.674 INFO:teuthology.orchestra.run.smithi071.stdout:Reading package lists... 2024-04-26T01:20:04.742 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T01:20:04.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:20:04.823 INFO:teuthology.orchestra.run.smithi045.stdout:Reading package lists... 2024-04-26T01:20:04.830 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-26T01:20:04.831 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-26T01:20:04.862 INFO:teuthology.orchestra.run.smithi181.stdout:Building dependency tree... 2024-04-26T01:20:04.862 INFO:teuthology.orchestra.run.smithi181.stdout:Reading state information... 2024-04-26T01:20:04.883 INFO:teuthology.orchestra.run.smithi071.stdout:Building dependency tree... 2024-04-26T01:20:04.884 INFO:teuthology.orchestra.run.smithi071.stdout:Reading state information... 2024-04-26T01:20:05.031 INFO:teuthology.orchestra.run.smithi045.stdout:Building dependency tree... 2024-04-26T01:20:05.032 INFO:teuthology.orchestra.run.smithi045.stdout:Reading state information... 2024-04-26T01:20:05.035 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:20:05.035 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:20:05.035 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:20:05.036 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:20:05.036 INFO:teuthology.orchestra.run.smithi037.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:20:05.036 INFO:teuthology.orchestra.run.smithi037.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:20:05.036 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:20:05.037 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-04-26T01:20:05.037 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:20:05.037 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.039 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet zip 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-04-26T01:20:05.040 INFO:teuthology.orchestra.run.smithi037.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-04-26T01:20:05.041 INFO:teuthology.orchestra.run.smithi037.stdout: btrfs-tools 2024-04-26T01:20:05.046 INFO:teuthology.orchestra.run.smithi181.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:20:05.047 INFO:teuthology.orchestra.run.smithi181.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:20:05.047 INFO:teuthology.orchestra.run.smithi181.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:20:05.047 INFO:teuthology.orchestra.run.smithi181.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:20:05.047 INFO:teuthology.orchestra.run.smithi181.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:20:05.047 INFO:teuthology.orchestra.run.smithi181.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:20:05.047 INFO:teuthology.orchestra.run.smithi181.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:20:05.048 INFO:teuthology.orchestra.run.smithi181.stdout:The following additional packages will be installed: 2024-04-26T01:20:05.048 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:20:05.048 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:20:05.049 INFO:teuthology.orchestra.run.smithi181.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:20:05.049 INFO:teuthology.orchestra.run.smithi181.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:20:05.050 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet zip 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout:Suggested packages: 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:20:05.051 INFO:teuthology.orchestra.run.smithi181.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:20:05.052 INFO:teuthology.orchestra.run.smithi181.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:20:05.052 INFO:teuthology.orchestra.run.smithi181.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:20:05.052 INFO:teuthology.orchestra.run.smithi181.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:20:05.052 INFO:teuthology.orchestra.run.smithi181.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:20:05.052 INFO:teuthology.orchestra.run.smithi181.stdout:Recommended packages: 2024-04-26T01:20:05.052 INFO:teuthology.orchestra.run.smithi181.stdout: btrfs-tools 2024-04-26T01:20:05.073 INFO:teuthology.orchestra.run.smithi071.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:20:05.073 INFO:teuthology.orchestra.run.smithi071.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:20:05.074 INFO:teuthology.orchestra.run.smithi071.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:20:05.074 INFO:teuthology.orchestra.run.smithi071.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:20:05.074 INFO:teuthology.orchestra.run.smithi071.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:20:05.075 INFO:teuthology.orchestra.run.smithi071.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:20:05.075 INFO:teuthology.orchestra.run.smithi071.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:20:05.075 INFO:teuthology.orchestra.run.smithi071.stdout:The following additional packages will be installed: 2024-04-26T01:20:05.075 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:20:05.075 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:20:05.075 INFO:teuthology.orchestra.run.smithi071.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:20:05.075 INFO:teuthology.orchestra.run.smithi071.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:20:05.076 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:20:05.076 INFO:teuthology.orchestra.run.smithi071.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:20:05.077 INFO:teuthology.orchestra.run.smithi071.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:20:05.077 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:20:05.077 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:20:05.077 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.077 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.077 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:20:05.078 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet zip 2024-04-26T01:20:05.079 INFO:teuthology.orchestra.run.smithi071.stdout:Suggested packages: 2024-04-26T01:20:05.079 INFO:teuthology.orchestra.run.smithi071.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:20:05.079 INFO:teuthology.orchestra.run.smithi071.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:20:05.079 INFO:teuthology.orchestra.run.smithi071.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:20:05.079 INFO:teuthology.orchestra.run.smithi071.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:20:05.079 INFO:teuthology.orchestra.run.smithi071.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:20:05.079 INFO:teuthology.orchestra.run.smithi071.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:20:05.080 INFO:teuthology.orchestra.run.smithi071.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:20:05.080 INFO:teuthology.orchestra.run.smithi071.stdout:Recommended packages: 2024-04-26T01:20:05.080 INFO:teuthology.orchestra.run.smithi071.stdout: btrfs-tools 2024-04-26T01:20:05.148 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-04-26T01:20:05.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:20:05.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:20:05.148 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:20:05.149 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:20:05.149 INFO:teuthology.orchestra.run.smithi037.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:20:05.149 INFO:teuthology.orchestra.run.smithi037.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.150 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:20:05.151 INFO:teuthology.orchestra.run.smithi037.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:20:05.152 INFO:teuthology.orchestra.run.smithi181.stdout:The following NEW packages will be installed: 2024-04-26T01:20:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:20:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:20:05.152 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:20:05.153 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.154 INFO:teuthology.orchestra.run.smithi181.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:20:05.155 INFO:teuthology.orchestra.run.smithi181.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:20:05.177 INFO:teuthology.orchestra.run.smithi071.stdout:The following NEW packages will be installed: 2024-04-26T01:20:05.177 INFO:teuthology.orchestra.run.smithi071.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:20:05.177 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:20:05.177 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:20:05.177 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:20:05.177 INFO:teuthology.orchestra.run.smithi071.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:20:05.177 INFO:teuthology.orchestra.run.smithi071.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:20:05.178 INFO:teuthology.orchestra.run.smithi071.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:20:05.178 INFO:teuthology.orchestra.run.smithi071.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:20:05.178 INFO:teuthology.orchestra.run.smithi071.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:20:05.178 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:20:05.179 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:20:05.180 INFO:teuthology.orchestra.run.smithi071.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:20:05.220 INFO:teuthology.orchestra.run.smithi181.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:20:05.220 INFO:teuthology.orchestra.run.smithi181.stdout:Need to get 179 MB of archives. 2024-04-26T01:20:05.220 INFO:teuthology.orchestra.run.smithi181.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:20:05.220 INFO:teuthology.orchestra.run.smithi181.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:20:05.236 INFO:teuthology.orchestra.run.smithi045.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T01:20:05.236 INFO:teuthology.orchestra.run.smithi045.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T01:20:05.237 INFO:teuthology.orchestra.run.smithi045.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T01:20:05.237 INFO:teuthology.orchestra.run.smithi045.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T01:20:05.237 INFO:teuthology.orchestra.run.smithi045.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T01:20:05.237 INFO:teuthology.orchestra.run.smithi045.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T01:20:05.237 INFO:teuthology.orchestra.run.smithi045.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T01:20:05.238 INFO:teuthology.orchestra.run.smithi045.stdout:The following additional packages will be installed: 2024-04-26T01:20:05.238 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T01:20:05.238 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T01:20:05.238 INFO:teuthology.orchestra.run.smithi045.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T01:20:05.239 INFO:teuthology.orchestra.run.smithi045.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.240 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet zip 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout:Suggested packages: 2024-04-26T01:20:05.241 INFO:teuthology.orchestra.run.smithi045.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout:Recommended packages: 2024-04-26T01:20:05.242 INFO:teuthology.orchestra.run.smithi045.stdout: btrfs-tools 2024-04-26T01:20:05.327 INFO:teuthology.orchestra.run.smithi181.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:20:05.331 INFO:teuthology.orchestra.run.smithi181.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:20:05.338 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:20:05.338 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 179 MB of archives. 2024-04-26T01:20:05.338 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:20:05.338 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:20:05.347 INFO:teuthology.orchestra.run.smithi045.stdout:The following NEW packages will be installed: 2024-04-26T01:20:05.347 INFO:teuthology.orchestra.run.smithi045.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T01:20:05.347 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T01:20:05.347 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T01:20:05.349 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T01:20:05.350 INFO:teuthology.orchestra.run.smithi045.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T01:20:05.366 INFO:teuthology.orchestra.run.smithi181.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:20:05.367 INFO:teuthology.orchestra.run.smithi071.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:20:05.367 INFO:teuthology.orchestra.run.smithi071.stdout:Need to get 179 MB of archives. 2024-04-26T01:20:05.367 INFO:teuthology.orchestra.run.smithi071.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:20:05.367 INFO:teuthology.orchestra.run.smithi071.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:20:05.401 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:20:05.408 INFO:teuthology.orchestra.run.smithi181.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:20:05.450 INFO:teuthology.orchestra.run.smithi071.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:20:05.545 INFO:teuthology.orchestra.run.smithi045.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T01:20:05.546 INFO:teuthology.orchestra.run.smithi045.stdout:Need to get 179 MB of archives. 2024-04-26T01:20:05.546 INFO:teuthology.orchestra.run.smithi045.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T01:20:05.546 INFO:teuthology.orchestra.run.smithi045.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T01:20:05.586 INFO:teuthology.orchestra.run.smithi181.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:20:05.658 INFO:teuthology.orchestra.run.smithi181.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:20:05.674 INFO:teuthology.orchestra.run.smithi181.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:20:05.681 INFO:teuthology.orchestra.run.smithi181.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:20:05.682 INFO:teuthology.orchestra.run.smithi181.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:20:05.683 INFO:teuthology.orchestra.run.smithi181.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:20:05.705 INFO:teuthology.orchestra.run.smithi181.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:20:05.712 INFO:teuthology.orchestra.run.smithi181.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:20:05.716 INFO:teuthology.orchestra.run.smithi045.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,541 kB] 2024-04-26T01:20:05.720 INFO:teuthology.orchestra.run.smithi181.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:20:05.723 INFO:teuthology.orchestra.run.smithi181.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:20:05.723 INFO:teuthology.orchestra.run.smithi181.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:20:05.737 INFO:teuthology.orchestra.run.smithi181.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:20:05.741 INFO:teuthology.orchestra.run.smithi181.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:20:05.745 INFO:teuthology.orchestra.run.smithi181.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:20:05.746 INFO:teuthology.orchestra.run.smithi181.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:20:05.762 INFO:teuthology.orchestra.run.smithi181.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:20:05.764 INFO:teuthology.orchestra.run.smithi181.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:20:05.764 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:20:05.764 INFO:teuthology.orchestra.run.smithi181.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:20:05.765 INFO:teuthology.orchestra.run.smithi181.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:20:05.765 INFO:teuthology.orchestra.run.smithi181.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:20:05.767 INFO:teuthology.orchestra.run.smithi181.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:20:05.776 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:20:05.783 INFO:teuthology.orchestra.run.smithi181.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:20:05.783 INFO:teuthology.orchestra.run.smithi181.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:20:05.789 INFO:teuthology.orchestra.run.smithi181.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:20:05.790 INFO:teuthology.orchestra.run.smithi181.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:20:05.792 INFO:teuthology.orchestra.run.smithi071.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:20:05.803 INFO:teuthology.orchestra.run.smithi181.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:20:05.804 INFO:teuthology.orchestra.run.smithi181.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:20:05.805 INFO:teuthology.orchestra.run.smithi071.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:20:05.806 INFO:teuthology.orchestra.run.smithi181.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:20:05.806 INFO:teuthology.orchestra.run.smithi181.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:20:05.807 INFO:teuthology.orchestra.run.smithi181.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:20:05.810 INFO:teuthology.orchestra.run.smithi181.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:20:05.824 INFO:teuthology.orchestra.run.smithi181.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:20:05.825 INFO:teuthology.orchestra.run.smithi181.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:20:05.838 INFO:teuthology.orchestra.run.smithi181.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:20:05.838 INFO:teuthology.orchestra.run.smithi181.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:20:05.847 INFO:teuthology.orchestra.run.smithi181.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:20:05.848 INFO:teuthology.orchestra.run.smithi181.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:20:05.851 INFO:teuthology.orchestra.run.smithi181.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:20:05.855 INFO:teuthology.orchestra.run.smithi181.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:20:05.856 INFO:teuthology.orchestra.run.smithi181.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:20:05.858 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:20:05.859 INFO:teuthology.orchestra.run.smithi181.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:20:05.872 INFO:teuthology.orchestra.run.smithi181.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:20:05.881 INFO:teuthology.orchestra.run.smithi181.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:20:05.882 INFO:teuthology.orchestra.run.smithi181.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:20:05.884 INFO:teuthology.orchestra.run.smithi181.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:20:05.886 INFO:teuthology.orchestra.run.smithi071.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:20:05.899 INFO:teuthology.orchestra.run.smithi181.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:20:05.971 INFO:teuthology.orchestra.run.smithi045.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T01:20:05.975 INFO:teuthology.orchestra.run.smithi181.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:20:05.976 INFO:teuthology.orchestra.run.smithi181.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:20:05.977 INFO:teuthology.orchestra.run.smithi181.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:20:05.977 INFO:teuthology.orchestra.run.smithi181.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:20:05.978 INFO:teuthology.orchestra.run.smithi181.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:20:05.984 INFO:teuthology.orchestra.run.smithi045.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T01:20:05.997 INFO:teuthology.orchestra.run.smithi181.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:20:05.999 INFO:teuthology.orchestra.run.smithi181.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:20:05.999 INFO:teuthology.orchestra.run.smithi181.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:20:06.024 INFO:teuthology.orchestra.run.smithi181.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:20:06.024 INFO:teuthology.orchestra.run.smithi181.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:20:06.031 INFO:teuthology.orchestra.run.smithi181.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:20:06.032 INFO:teuthology.orchestra.run.smithi181.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:20:06.032 INFO:teuthology.orchestra.run.smithi181.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:20:06.033 INFO:teuthology.orchestra.run.smithi181.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:20:06.033 INFO:teuthology.orchestra.run.smithi181.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:20:06.034 INFO:teuthology.orchestra.run.smithi181.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:20:06.037 INFO:teuthology.orchestra.run.smithi181.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:20:06.052 INFO:teuthology.orchestra.run.smithi181.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:20:06.054 INFO:teuthology.orchestra.run.smithi181.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:20:06.054 INFO:teuthology.orchestra.run.smithi181.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:20:06.065 INFO:teuthology.orchestra.run.smithi045.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T01:20:06.073 INFO:teuthology.orchestra.run.smithi181.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:20:06.076 INFO:teuthology.orchestra.run.smithi181.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:20:06.077 INFO:teuthology.orchestra.run.smithi181.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:20:06.078 INFO:teuthology.orchestra.run.smithi181.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:20:06.084 INFO:teuthology.orchestra.run.smithi181.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:20:06.085 INFO:teuthology.orchestra.run.smithi181.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:20:06.088 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:20:06.089 INFO:teuthology.orchestra.run.smithi181.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:20:06.095 INFO:teuthology.orchestra.run.smithi181.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:20:06.096 INFO:teuthology.orchestra.run.smithi181.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:20:06.096 INFO:teuthology.orchestra.run.smithi181.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:20:06.100 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:20:06.116 INFO:teuthology.orchestra.run.smithi181.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:20:06.116 INFO:teuthology.orchestra.run.smithi071.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:20:06.117 INFO:teuthology.orchestra.run.smithi181.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:20:06.127 INFO:teuthology.orchestra.run.smithi181.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:20:06.128 INFO:teuthology.orchestra.run.smithi071.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:20:06.128 INFO:teuthology.orchestra.run.smithi181.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:20:06.128 INFO:teuthology.orchestra.run.smithi181.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:20:06.129 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:20:06.131 INFO:teuthology.orchestra.run.smithi181.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:20:06.132 INFO:teuthology.orchestra.run.smithi181.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:20:06.139 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:20:06.140 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:20:06.141 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:20:06.158 INFO:teuthology.orchestra.run.smithi071.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:20:06.160 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:20:06.167 INFO:teuthology.orchestra.run.smithi071.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:20:06.168 INFO:teuthology.orchestra.run.smithi071.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:20:06.169 INFO:teuthology.orchestra.run.smithi071.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:20:06.188 INFO:teuthology.orchestra.run.smithi071.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:20:06.195 INFO:teuthology.orchestra.run.smithi045.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:20:06.296 INFO:teuthology.orchestra.run.smithi045.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T01:20:06.308 INFO:teuthology.orchestra.run.smithi045.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T01:20:06.328 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:20:06.338 INFO:teuthology.orchestra.run.smithi045.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T01:20:06.347 INFO:teuthology.orchestra.run.smithi045.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T01:20:06.348 INFO:teuthology.orchestra.run.smithi045.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T01:20:06.349 INFO:teuthology.orchestra.run.smithi045.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T01:20:06.356 INFO:teuthology.orchestra.run.smithi071.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:20:06.368 INFO:teuthology.orchestra.run.smithi045.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T01:20:06.408 INFO:teuthology.orchestra.run.smithi045.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:20:06.451 INFO:teuthology.orchestra.run.smithi045.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:20:06.458 INFO:teuthology.orchestra.run.smithi045.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:20:06.459 INFO:teuthology.orchestra.run.smithi045.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:20:06.464 INFO:teuthology.orchestra.run.smithi045.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:20:06.484 INFO:teuthology.orchestra.run.smithi045.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:20:06.491 INFO:teuthology.orchestra.run.smithi045.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:20:06.537 INFO:teuthology.orchestra.run.smithi045.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T01:20:06.738 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:20:06.746 INFO:teuthology.orchestra.run.smithi181.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:20:06.759 INFO:teuthology.orchestra.run.smithi071.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:20:06.771 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:20:06.773 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:20:06.794 INFO:teuthology.orchestra.run.smithi071.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:20:06.797 INFO:teuthology.orchestra.run.smithi071.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:20:06.826 INFO:teuthology.orchestra.run.smithi045.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:20:06.829 INFO:teuthology.orchestra.run.smithi045.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:20:06.845 INFO:teuthology.orchestra.run.smithi045.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:20:06.854 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:20:06.874 INFO:teuthology.orchestra.run.smithi071.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:20:06.904 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:20:06.927 INFO:teuthology.orchestra.run.smithi071.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:20:06.927 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:20:06.933 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:20:06.941 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:20:06.946 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:20:06.948 INFO:teuthology.orchestra.run.smithi071.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:20:06.949 INFO:teuthology.orchestra.run.smithi045.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T01:20:06.950 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:20:06.954 INFO:teuthology.orchestra.run.smithi071.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:20:06.955 INFO:teuthology.orchestra.run.smithi071.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:20:06.967 INFO:teuthology.orchestra.run.smithi071.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:20:06.968 INFO:teuthology.orchestra.run.smithi071.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:20:06.980 INFO:teuthology.orchestra.run.smithi045.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T01:20:06.981 INFO:teuthology.orchestra.run.smithi045.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T01:20:07.063 INFO:teuthology.orchestra.run.smithi045.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T01:20:07.112 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:20:07.113 INFO:teuthology.orchestra.run.smithi045.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T01:20:07.133 INFO:teuthology.orchestra.run.smithi071.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:20:07.134 INFO:teuthology.orchestra.run.smithi045.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T01:20:07.140 INFO:teuthology.orchestra.run.smithi045.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T01:20:07.142 INFO:teuthology.orchestra.run.smithi045.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T01:20:07.154 INFO:teuthology.orchestra.run.smithi045.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T01:20:07.155 INFO:teuthology.orchestra.run.smithi045.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T01:20:07.196 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:20:07.217 INFO:teuthology.orchestra.run.smithi071.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:20:07.224 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:20:07.241 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:20:07.246 INFO:teuthology.orchestra.run.smithi071.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:20:07.267 INFO:teuthology.orchestra.run.smithi071.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:20:07.289 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:20:07.310 INFO:teuthology.orchestra.run.smithi071.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:20:07.323 INFO:teuthology.orchestra.run.smithi045.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T01:20:07.351 INFO:teuthology.orchestra.run.smithi181.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:20:07.407 INFO:teuthology.orchestra.run.smithi045.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T01:20:07.436 INFO:teuthology.orchestra.run.smithi045.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T01:20:07.452 INFO:teuthology.orchestra.run.smithi045.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T01:20:07.485 INFO:teuthology.orchestra.run.smithi181.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:20:07.497 INFO:teuthology.orchestra.run.smithi045.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T01:20:07.505 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:20:07.521 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:20:07.525 INFO:teuthology.orchestra.run.smithi071.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:20:07.527 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:20:07.533 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:20:07.540 INFO:teuthology.orchestra.run.smithi071.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:20:07.547 INFO:teuthology.orchestra.run.smithi071.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:20:07.549 INFO:teuthology.orchestra.run.smithi181.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:20:07.551 INFO:teuthology.orchestra.run.smithi181.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:20:07.552 INFO:teuthology.orchestra.run.smithi071.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:20:07.555 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:20:07.558 INFO:teuthology.orchestra.run.smithi181.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:20:07.559 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:20:07.561 INFO:teuthology.orchestra.run.smithi181.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:20:07.574 INFO:teuthology.orchestra.run.smithi071.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:20:07.578 INFO:teuthology.orchestra.run.smithi071.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:20:07.635 INFO:teuthology.orchestra.run.smithi071.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:20:07.680 INFO:teuthology.orchestra.run.smithi181.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:20:07.713 INFO:teuthology.orchestra.run.smithi045.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T01:20:07.725 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:20:07.729 INFO:teuthology.orchestra.run.smithi045.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T01:20:07.736 INFO:teuthology.orchestra.run.smithi045.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T01:20:07.741 INFO:teuthology.orchestra.run.smithi045.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T01:20:07.745 INFO:teuthology.orchestra.run.smithi071.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:20:07.762 INFO:teuthology.orchestra.run.smithi045.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T01:20:07.766 INFO:teuthology.orchestra.run.smithi045.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T01:20:07.932 INFO:teuthology.orchestra.run.smithi045.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T01:20:07.933 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-26T01:20:08.042 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:20:08.079 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:20:08.080 INFO:teuthology.orchestra.run.smithi071.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:20:08.090 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:20:08.111 INFO:teuthology.orchestra.run.smithi071.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:20:08.122 INFO:teuthology.orchestra.run.smithi071.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:20:08.139 INFO:teuthology.orchestra.run.smithi045.stdout:Get:47 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:20:08.270 INFO:teuthology.orchestra.run.smithi045.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T01:20:08.294 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:20:08.296 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:20:08.301 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:20:08.307 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:20:08.323 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:20:08.324 INFO:teuthology.orchestra.run.smithi071.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:20:08.326 INFO:teuthology.orchestra.run.smithi071.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:20:08.331 INFO:teuthology.orchestra.run.smithi071.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:20:08.337 INFO:teuthology.orchestra.run.smithi071.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:20:08.343 INFO:teuthology.orchestra.run.smithi037.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:20:08.349 INFO:teuthology.orchestra.run.smithi037.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:20:08.352 INFO:teuthology.orchestra.run.smithi071.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:20:08.365 INFO:teuthology.orchestra.run.smithi045.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:20:08.370 INFO:teuthology.orchestra.run.smithi045.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T01:20:08.374 INFO:teuthology.orchestra.run.smithi071.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:20:08.378 INFO:teuthology.orchestra.run.smithi045.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:20:08.380 INFO:teuthology.orchestra.run.smithi071.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:20:08.382 INFO:teuthology.orchestra.run.smithi045.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:20:08.397 INFO:teuthology.orchestra.run.smithi045.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T01:20:08.524 INFO:teuthology.orchestra.run.smithi045.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:20:08.524 INFO:teuthology.orchestra.run.smithi037.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:20:08.556 INFO:teuthology.orchestra.run.smithi071.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:20:08.787 INFO:teuthology.orchestra.run.smithi037.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:20:08.819 INFO:teuthology.orchestra.run.smithi071.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:20:08.848 INFO:teuthology.orchestra.run.smithi181.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:20:08.865 INFO:teuthology.orchestra.run.smithi181.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:20:08.965 INFO:teuthology.orchestra.run.smithi181.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:20:08.966 INFO:teuthology.orchestra.run.smithi037.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:20:08.979 INFO:teuthology.orchestra.run.smithi037.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:20:08.997 INFO:teuthology.orchestra.run.smithi071.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:20:08.999 INFO:teuthology.orchestra.run.smithi037.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:20:09.011 INFO:teuthology.orchestra.run.smithi071.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:20:09.027 INFO:teuthology.orchestra.run.smithi045.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:20:09.031 INFO:teuthology.orchestra.run.smithi071.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:20:09.064 INFO:teuthology.orchestra.run.smithi037.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:20:09.095 INFO:teuthology.orchestra.run.smithi071.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:20:09.149 INFO:teuthology.orchestra.run.smithi045.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T01:20:09.154 INFO:teuthology.orchestra.run.smithi045.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T01:20:09.180 INFO:teuthology.orchestra.run.smithi045.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T01:20:09.206 INFO:teuthology.orchestra.run.smithi045.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T01:20:09.293 INFO:teuthology.orchestra.run.smithi045.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T01:20:09.351 INFO:teuthology.orchestra.run.smithi037.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:20:09.353 INFO:teuthology.orchestra.run.smithi037.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:20:09.356 INFO:teuthology.orchestra.run.smithi037.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:20:09.356 INFO:teuthology.orchestra.run.smithi037.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:20:09.358 INFO:teuthology.orchestra.run.smithi037.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:20:09.391 INFO:teuthology.orchestra.run.smithi071.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:20:09.393 INFO:teuthology.orchestra.run.smithi071.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:20:09.395 INFO:teuthology.orchestra.run.smithi071.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:20:09.396 INFO:teuthology.orchestra.run.smithi071.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:20:09.399 INFO:teuthology.orchestra.run.smithi071.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:20:09.410 INFO:teuthology.orchestra.run.smithi045.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T01:20:09.450 INFO:teuthology.orchestra.run.smithi071.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:20:09.454 INFO:teuthology.orchestra.run.smithi037.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-26T01:20:09.539 INFO:teuthology.orchestra.run.smithi045.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T01:20:09.587 INFO:teuthology.orchestra.run.smithi037.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:20:09.618 INFO:teuthology.orchestra.run.smithi071.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:20:09.696 INFO:teuthology.orchestra.run.smithi037.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:20:09.754 INFO:teuthology.orchestra.run.smithi071.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-26T01:20:09.817 INFO:teuthology.orchestra.run.smithi037.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:20:09.842 INFO:teuthology.orchestra.run.smithi037.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:20:09.890 INFO:teuthology.orchestra.run.smithi071.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-26T01:20:09.896 INFO:teuthology.orchestra.run.smithi045.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T01:20:09.896 INFO:teuthology.orchestra.run.smithi037.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:20:09.914 INFO:teuthology.orchestra.run.smithi071.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-26T01:20:09.926 INFO:teuthology.orchestra.run.smithi037.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:20:09.982 INFO:teuthology.orchestra.run.smithi071.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.8 kB] 2024-04-26T01:20:10.000 INFO:teuthology.orchestra.run.smithi071.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-26T01:20:10.017 INFO:teuthology.orchestra.run.smithi037.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:20:10.023 INFO:teuthology.orchestra.run.smithi037.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:20:10.025 INFO:teuthology.orchestra.run.smithi037.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:20:10.056 INFO:teuthology.orchestra.run.smithi071.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:20:10.062 INFO:teuthology.orchestra.run.smithi071.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:20:10.141 INFO:teuthology.orchestra.run.smithi071.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-26T01:20:10.161 INFO:teuthology.orchestra.run.smithi045.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T01:20:10.312 INFO:teuthology.orchestra.run.smithi037.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:20:10.313 INFO:teuthology.orchestra.run.smithi037.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:20:10.315 INFO:teuthology.orchestra.run.smithi037.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:20:10.317 INFO:teuthology.orchestra.run.smithi037.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:20:10.318 INFO:teuthology.orchestra.run.smithi037.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:20:10.320 INFO:teuthology.orchestra.run.smithi037.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:20:10.321 INFO:teuthology.orchestra.run.smithi037.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:20:10.323 INFO:teuthology.orchestra.run.smithi037.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:20:10.348 INFO:teuthology.orchestra.run.smithi045.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T01:20:10.353 INFO:teuthology.orchestra.run.smithi071.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:20:10.354 INFO:teuthology.orchestra.run.smithi071.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:20:10.356 INFO:teuthology.orchestra.run.smithi071.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:20:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:20:10.359 INFO:teuthology.orchestra.run.smithi071.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:20:10.362 INFO:teuthology.orchestra.run.smithi071.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:20:10.363 INFO:teuthology.orchestra.run.smithi071.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:20:10.365 INFO:teuthology.orchestra.run.smithi071.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:20:10.406 INFO:teuthology.orchestra.run.smithi045.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T01:20:10.438 INFO:teuthology.orchestra.run.smithi045.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:20:10.439 INFO:teuthology.orchestra.run.smithi045.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T01:20:10.439 INFO:teuthology.orchestra.run.smithi045.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:20:10.443 INFO:teuthology.orchestra.run.smithi045.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T01:20:10.456 INFO:teuthology.orchestra.run.smithi045.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:20:10.504 INFO:teuthology.orchestra.run.smithi037.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:20:10.546 INFO:teuthology.orchestra.run.smithi071.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:20:10.562 INFO:teuthology.orchestra.run.smithi045.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:20:10.606 INFO:teuthology.orchestra.run.smithi045.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:20:10.611 INFO:teuthology.orchestra.run.smithi045.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:20:10.747 INFO:teuthology.orchestra.run.smithi045.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:20:10.780 INFO:teuthology.orchestra.run.smithi045.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T01:20:10.781 INFO:teuthology.orchestra.run.smithi045.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T01:20:10.784 INFO:teuthology.orchestra.run.smithi045.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T01:20:10.785 INFO:teuthology.orchestra.run.smithi045.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T01:20:10.786 INFO:teuthology.orchestra.run.smithi045.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T01:20:10.920 INFO:teuthology.orchestra.run.smithi037.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:20:10.953 INFO:teuthology.orchestra.run.smithi071.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:20:10.966 INFO:teuthology.orchestra.run.smithi037.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:20:10.999 INFO:teuthology.orchestra.run.smithi071.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:20:11.001 INFO:teuthology.orchestra.run.smithi037.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:20:11.013 INFO:teuthology.orchestra.run.smithi045.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T01:20:11.018 INFO:teuthology.orchestra.run.smithi071.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:20:11.098 INFO:teuthology.orchestra.run.smithi037.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:20:11.131 INFO:teuthology.orchestra.run.smithi071.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:20:11.138 INFO:teuthology.orchestra.run.smithi037.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:20:11.151 INFO:teuthology.orchestra.run.smithi037.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:20:11.154 INFO:teuthology.orchestra.run.smithi037.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:20:11.171 INFO:teuthology.orchestra.run.smithi071.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:20:11.182 INFO:teuthology.orchestra.run.smithi071.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:20:11.187 INFO:teuthology.orchestra.run.smithi071.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:20:11.201 INFO:teuthology.orchestra.run.smithi037.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:20:11.205 INFO:teuthology.orchestra.run.smithi037.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:20:11.226 INFO:teuthology.orchestra.run.smithi037.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:20:11.235 INFO:teuthology.orchestra.run.smithi071.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:20:11.238 INFO:teuthology.orchestra.run.smithi071.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:20:11.260 INFO:teuthology.orchestra.run.smithi071.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:20:11.272 INFO:teuthology.orchestra.run.smithi045.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:20:11.273 INFO:teuthology.orchestra.run.smithi045.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:20:11.390 INFO:teuthology.orchestra.run.smithi037.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:20:11.424 INFO:teuthology.orchestra.run.smithi071.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:20:11.454 INFO:teuthology.orchestra.run.smithi045.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T01:20:11.460 INFO:teuthology.orchestra.run.smithi045.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T01:20:11.657 INFO:teuthology.orchestra.run.smithi037.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:20:11.674 INFO:teuthology.orchestra.run.smithi037.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:20:11.697 INFO:teuthology.orchestra.run.smithi071.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:20:11.715 INFO:teuthology.orchestra.run.smithi071.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:20:11.750 INFO:teuthology.orchestra.run.smithi045.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T01:20:11.750 INFO:teuthology.orchestra.run.smithi045.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T01:20:11.753 INFO:teuthology.orchestra.run.smithi045.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T01:20:11.755 INFO:teuthology.orchestra.run.smithi045.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T01:20:11.757 INFO:teuthology.orchestra.run.smithi045.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T01:20:11.762 INFO:teuthology.orchestra.run.smithi045.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T01:20:11.763 INFO:teuthology.orchestra.run.smithi045.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T01:20:11.763 INFO:teuthology.orchestra.run.smithi045.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T01:20:11.828 INFO:teuthology.orchestra.run.smithi037.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:20:11.857 INFO:teuthology.orchestra.run.smithi037.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:20:11.869 INFO:teuthology.orchestra.run.smithi071.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:20:11.899 INFO:teuthology.orchestra.run.smithi071.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:20:11.962 INFO:teuthology.orchestra.run.smithi045.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T01:20:12.030 INFO:teuthology.orchestra.run.smithi037.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:20:12.033 INFO:teuthology.orchestra.run.smithi037.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:20:12.034 INFO:teuthology.orchestra.run.smithi037.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:20:12.055 INFO:teuthology.orchestra.run.smithi037.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:20:12.056 INFO:teuthology.orchestra.run.smithi071.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:20:12.058 INFO:teuthology.orchestra.run.smithi071.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:20:12.060 INFO:teuthology.orchestra.run.smithi037.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:20:12.060 INFO:teuthology.orchestra.run.smithi071.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:20:12.082 INFO:teuthology.orchestra.run.smithi071.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:20:12.087 INFO:teuthology.orchestra.run.smithi071.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:20:12.118 INFO:teuthology.orchestra.run.smithi037.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:20:12.136 INFO:teuthology.orchestra.run.smithi037.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:20:12.271 INFO:teuthology.orchestra.run.smithi037.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:20:12.503 INFO:teuthology.orchestra.run.smithi045.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T01:20:12.595 INFO:teuthology.orchestra.run.smithi045.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T01:20:12.645 INFO:teuthology.orchestra.run.smithi071.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-26T01:20:12.691 INFO:teuthology.orchestra.run.smithi181.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:20:12.703 INFO:teuthology.orchestra.run.smithi071.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [467 kB] 2024-04-26T01:20:12.756 INFO:teuthology.orchestra.run.smithi045.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T01:20:12.945 INFO:teuthology.orchestra.run.smithi071.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-26T01:20:13.398 INFO:teuthology.orchestra.run.smithi045.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T01:20:13.790 INFO:teuthology.orchestra.run.smithi181.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:20:13.905 INFO:teuthology.orchestra.run.smithi045.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T01:20:13.913 INFO:teuthology.orchestra.run.smithi181.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:20:14.040 INFO:teuthology.orchestra.run.smithi045.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T01:20:14.092 INFO:teuthology.orchestra.run.smithi045.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T01:20:14.119 INFO:teuthology.orchestra.run.smithi181.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:20:14.671 INFO:teuthology.orchestra.run.smithi181.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:20:15.148 INFO:teuthology.orchestra.run.smithi045.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T01:20:15.253 INFO:teuthology.orchestra.run.smithi045.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T01:20:15.937 INFO:teuthology.orchestra.run.smithi045.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T01:20:16.111 INFO:teuthology.orchestra.run.smithi045.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T01:20:16.391 INFO:teuthology.orchestra.run.smithi045.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T01:20:16.420 INFO:teuthology.orchestra.run.smithi045.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T01:20:16.630 INFO:teuthology.orchestra.run.smithi045.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T01:20:16.748 INFO:teuthology.orchestra.run.smithi045.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T01:20:17.041 INFO:teuthology.orchestra.run.smithi181.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:20:17.911 INFO:teuthology.orchestra.run.smithi045.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T01:20:17.929 INFO:teuthology.orchestra.run.smithi045.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T01:20:17.951 INFO:teuthology.orchestra.run.smithi045.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T01:20:18.256 INFO:teuthology.orchestra.run.smithi045.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T01:20:18.320 INFO:teuthology.orchestra.run.smithi045.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T01:20:19.030 INFO:teuthology.orchestra.run.smithi045.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:20:19.032 INFO:teuthology.orchestra.run.smithi045.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:20:19.033 INFO:teuthology.orchestra.run.smithi045.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:20:22.546 INFO:teuthology.orchestra.run.smithi045.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:20:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:Fetched 179 MB in 17s (10.4 MB/s) 2024-04-26T01:20:23.258 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:20:23.358 INFO:teuthology.orchestra.run.smithi045.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:20:23.364 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:20:23.432 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:20:23.718 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:20:23.745 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:20:23.777 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:20:24.113 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:20:24.140 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:20:24.179 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:20:24.633 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:20:24.660 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:24.707 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:25.077 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:20:25.104 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:25.135 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:25.505 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:20:25.532 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:25.580 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:25.941 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:20:25.968 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:20:26.008 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:20:26.594 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librados2. 2024-04-26T01:20:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:26.660 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:27.105 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:20:27.132 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:27.171 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:27.253 INFO:teuthology.orchestra.run.smithi037.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:20:27.304 INFO:teuthology.orchestra.run.smithi181.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:20:27.305 INFO:teuthology.orchestra.run.smithi181.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:20:27.527 INFO:teuthology.orchestra.run.smithi181.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:20:27.616 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:20:27.642 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:27.682 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:27.968 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:20:27.995 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:28.034 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:28.324 INFO:teuthology.orchestra.run.smithi181.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:20:28.337 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:20:28.353 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:28.387 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:28.606 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:20:28.622 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:28.635 INFO:teuthology.orchestra.run.smithi181.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:20:28.657 INFO:teuthology.orchestra.run.smithi181.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:20:28.664 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:28.940 INFO:teuthology.orchestra.run.smithi071.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-26T01:20:28.959 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:20:28.985 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:29.025 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:29.130 INFO:teuthology.orchestra.run.smithi037.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:20:29.177 INFO:teuthology.orchestra.run.smithi037.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:20:29.227 INFO:teuthology.orchestra.run.smithi037.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:20:29.269 INFO:teuthology.orchestra.run.smithi181.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:20:29.353 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:20:29.380 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:20:29.420 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:20:29.718 INFO:teuthology.orchestra.run.smithi037.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:20:29.747 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:20:29.774 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:20:29.814 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:20:29.882 INFO:teuthology.orchestra.run.smithi071.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-26T01:20:29.918 INFO:teuthology.orchestra.run.smithi071.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-26T01:20:29.947 INFO:teuthology.orchestra.run.smithi071.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,526 kB] 2024-04-26T01:20:30.083 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:20:30.099 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:30.133 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:30.304 INFO:teuthology.orchestra.run.smithi071.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-26T01:20:30.411 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:20:30.434 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:20:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:20:30.772 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:20:30.787 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:20:30.830 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:20:31.036 INFO:teuthology.orchestra.run.smithi181.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:20:31.037 INFO:teuthology.orchestra.run.smithi181.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:20:31.157 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:20:31.173 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:20:31.216 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:20:31.385 INFO:teuthology.orchestra.run.smithi071.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:20:31.438 INFO:teuthology.orchestra.run.smithi037.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-26T01:20:31.519 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:20:31.545 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:20:31.585 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:20:31.949 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:20:31.976 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:20:32.147 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:20:32.399 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package zip. 2024-04-26T01:20:32.415 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:20:32.457 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:20:32.777 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package unzip. 2024-04-26T01:20:32.804 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:20:32.843 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:20:33.206 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:20:33.222 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:20:33.255 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:20:33.638 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:20:33.665 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:33.704 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:34.273 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:20:34.300 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:34.340 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:34.401 INFO:teuthology.orchestra.run.smithi071.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:20:34.402 INFO:teuthology.orchestra.run.smithi071.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:20:34.473 INFO:teuthology.orchestra.run.smithi071.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:20:34.684 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:20:34.711 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:20:34.751 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:20:34.763 INFO:teuthology.orchestra.run.smithi071.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:20:34.839 INFO:teuthology.orchestra.run.smithi071.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:20:34.871 INFO:teuthology.orchestra.run.smithi071.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:20:35.011 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:20:35.027 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:35.061 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:35.130 INFO:teuthology.orchestra.run.smithi071.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:20:35.490 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:20:35.517 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:35.540 INFO:teuthology.orchestra.run.smithi037.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-26T01:20:35.541 INFO:teuthology.orchestra.run.smithi037.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-26T01:20:35.556 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:35.627 INFO:teuthology.orchestra.run.smithi037.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-26T01:20:35.982 INFO:teuthology.orchestra.run.smithi037.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [791 kB] 2024-04-26T01:20:36.090 INFO:teuthology.orchestra.run.smithi037.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-26T01:20:36.121 INFO:teuthology.orchestra.run.smithi037.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-26T01:20:36.166 INFO:teuthology.orchestra.run.smithi071.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:20:36.166 INFO:teuthology.orchestra.run.smithi071.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:20:36.428 INFO:teuthology.orchestra.run.smithi037.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-26T01:20:37.685 INFO:teuthology.orchestra.run.smithi037.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-26T01:20:37.685 INFO:teuthology.orchestra.run.smithi037.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-26T01:20:37.954 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:20:37.981 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:38.030 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:38.070 INFO:teuthology.orchestra.run.smithi181.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:20:38.156 INFO:teuthology.orchestra.run.smithi181.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:20:38.158 INFO:teuthology.orchestra.run.smithi181.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:20:38.574 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:20:38.601 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:20:38.640 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:20:38.935 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:20:38.961 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:20:39.002 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:20:39.338 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:20:39.364 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:20:39.404 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:20:39.506 INFO:teuthology.orchestra.run.smithi181.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:20:39.648 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:20:39.675 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:20:39.707 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:20:39.984 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:20:40.014 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:20:40.051 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:20:40.070 INFO:teuthology.orchestra.run.smithi181.stdout:Fetched 179 MB in 34s (5,226 kB/s) 2024-04-26T01:20:40.233 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:20:40.333 INFO:teuthology.orchestra.run.smithi181.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:20:40.339 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:20:40.353 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:20:40.380 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:20:40.422 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:20:40.431 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:20:40.723 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:20:40.750 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:20:40.777 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:20:40.789 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:20:40.803 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:20:40.843 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:20:41.050 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:20:41.074 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:20:41.108 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:20:41.121 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:20:41.147 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:20:41.187 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:20:41.403 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:20:41.430 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:20:41.469 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:20:41.616 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:20:41.642 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:41.691 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:41.800 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:20:41.827 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:20:41.858 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:20:42.086 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:20:42.112 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:42.144 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:20:42.152 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:42.171 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:20:42.180 INFO:teuthology.orchestra.run.smithi071.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:20:42.183 INFO:teuthology.orchestra.run.smithi071.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:20:42.184 INFO:teuthology.orchestra.run.smithi071.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:20:42.210 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:20:42.480 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:20:42.506 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:42.513 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:20:42.529 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:20:42.547 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:42.572 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:20:42.875 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:20:42.893 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:20:42.901 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:20:42.919 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:20:42.958 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:20:42.966 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:20:43.277 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:20:43.303 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librados2. 2024-04-26T01:20:43.304 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:20:43.330 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:43.344 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:20:43.369 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:43.571 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:20:43.598 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:20:43.638 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:20:43.831 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:20:43.857 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:43.897 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:43.941 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:20:43.968 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:20:44.007 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:20:44.141 INFO:teuthology.orchestra.run.smithi071.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:20:44.308 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:20:44.335 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:44.368 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:20:44.375 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:44.395 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:20:44.438 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:20:44.661 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:20:44.676 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:44.705 INFO:teuthology.orchestra.run.smithi071.stdout:Fetched 179 MB in 39s (4,607 kB/s) 2024-04-26T01:20:44.719 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:44.746 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:20:44.773 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:20:44.812 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:20:44.863 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:20:44.962 INFO:teuthology.orchestra.run.smithi071.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:20:44.968 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:20:45.022 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:20:45.048 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:45.053 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:20:45.088 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:45.115 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:20:45.142 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:20:45.182 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:20:45.204 INFO:teuthology.orchestra.run.smithi037.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-26T01:20:45.206 INFO:teuthology.orchestra.run.smithi037.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-26T01:20:45.208 INFO:teuthology.orchestra.run.smithi037.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-26T01:20:45.324 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:20:45.340 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:45.374 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:45.473 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:20:45.499 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:20:45.518 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:20:45.534 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:20:45.548 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:20:45.576 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:20:45.685 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:20:45.712 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:45.752 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:45.879 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:20:45.906 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:20:45.945 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:20:45.951 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:20:45.977 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:20:46.026 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:20:46.071 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:20:46.098 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:20:46.138 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:20:46.207 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:20:46.223 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:20:46.256 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:20:46.458 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:20:46.479 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:20:46.484 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:20:46.505 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:46.524 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:20:46.526 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:20:46.553 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:20:46.554 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:46.592 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:20:46.654 INFO:teuthology.orchestra.run.smithi037.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.4 kB] 2024-04-26T01:20:46.768 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:20:46.784 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:46.818 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:46.989 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:20:47.016 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:20:47.032 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:20:47.059 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:47.108 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:47.180 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:20:47.207 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:20:47.246 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:20:47.249 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 179 MB in 41s (4,325 kB/s) 2024-04-26T01:20:47.255 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:20:47.486 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T01:20:47.518 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:20:47.524 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:20:47.545 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:47.551 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:20:47.590 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T01:20:47.595 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:47.595 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:20:47.596 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:20:47.596 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T01:20:47.618 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:20:47.658 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:20:47.667 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:20:47.893 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:20:47.909 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:20:47.939 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:20:47.952 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:20:47.954 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T01:20:47.965 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:20:47.981 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T01:20:48.011 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:20:48.012 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:20:48.013 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:20:48.037 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:20:48.086 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:20:48.246 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:20:48.273 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:20:48.305 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:20:48.374 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T01:20:48.391 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librados2. 2024-04-26T01:20:48.401 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T01:20:48.418 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:48.449 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:20:48.455 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:20:48.466 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:48.482 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:20:48.530 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:20:48.625 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:20:48.640 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:20:48.683 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:20:48.877 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T01:20:48.905 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:48.925 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:20:48.935 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package zip. 2024-04-26T01:20:48.951 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:20:48.952 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:48.961 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:20:48.985 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:20:48.999 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:20:49.002 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:20:49.011 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:49.060 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:49.263 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package unzip. 2024-04-26T01:20:49.279 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:20:49.311 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:20:49.313 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:20:49.338 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:49.363 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T01:20:49.385 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:49.390 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:49.430 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:49.692 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:20:49.719 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:20:49.722 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:20:49.748 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:49.759 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:20:49.767 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T01:20:49.782 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:20:49.788 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:49.793 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T01:20:49.809 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:49.833 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:20:49.848 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:50.074 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:20:50.100 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:50.133 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:20:50.140 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:50.160 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:50.168 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:20:50.195 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:20:50.195 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T01:20:50.199 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:50.222 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T01:20:50.234 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:20:50.261 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:20:50.520 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:20:50.535 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:20:50.548 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:20:50.561 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:50.587 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:20:50.589 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librados2. 2024-04-26T01:20:50.610 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:50.616 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../007-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:50.656 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:50.735 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:20:50.751 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:50.794 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:50.906 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:20:50.912 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:20:50.933 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:50.938 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:50.964 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:50.987 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:51.059 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librbd1. 2024-04-26T01:20:51.074 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../008-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:51.108 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:51.113 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:20:51.140 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:20:51.171 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:20:51.315 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:20:51.341 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:51.351 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:20:51.378 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:51.381 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:51.417 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:51.475 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:20:51.501 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:51.541 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:51.553 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T01:20:51.580 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:51.644 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:51.685 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:20:51.711 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:20:51.751 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:20:51.930 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rados. 2024-04-26T01:20:51.946 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:51.961 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:20:51.988 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:51.988 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:52.027 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:52.045 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:20:52.072 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:52.112 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:52.146 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:20:52.172 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:20:52.220 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:20:52.292 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T01:20:52.318 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:52.350 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:52.540 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:20:52.544 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T01:20:52.560 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:52.566 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:52.594 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:52.615 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:52.897 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T01:20:52.923 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:52.963 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:52.993 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:20:53.019 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:20:53.059 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:20:53.216 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph. 2024-04-26T01:20:53.243 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:53.282 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:53.283 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T01:20:53.310 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T01:20:53.316 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:20:53.337 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:20:53.344 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:53.350 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:20:53.352 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:20:53.383 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:53.395 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:20:53.609 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:20:53.637 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:53.669 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T01:20:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:53.696 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T01:20:53.736 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T01:20:53.740 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:20:53.766 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:20:53.815 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:20:53.945 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:20:53.963 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T01:20:53.972 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:20:53.979 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:54.012 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:20:54.013 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:54.062 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:20:54.089 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:54.129 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:54.134 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:20:54.161 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:20:54.201 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:20:54.298 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:20:54.324 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:20:54.364 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:20:54.366 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T01:20:54.393 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T01:20:54.433 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:20:54.487 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:20:54.490 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:20:54.513 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:20:54.517 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:54.554 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:20:54.556 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:54.710 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T01:20:54.726 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:20:54.727 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T01:20:54.753 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:20:54.769 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:20:54.790 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package zip. 2024-04-26T01:20:54.792 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:20:54.816 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:20:54.862 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:20:54.867 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:20:54.894 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:20:54.934 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:20:55.054 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:20:55.063 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T01:20:55.080 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T01:20:55.080 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:20:55.113 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:20:55.120 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:20:55.245 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:20:55.272 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:20:55.276 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package unzip. 2024-04-26T01:20:55.302 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:20:55.311 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:20:55.343 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:20:55.423 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:20:55.442 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua5.1. 2024-04-26T01:20:55.450 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:20:55.468 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T01:20:55.490 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:20:55.508 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:20:55.680 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:20:55.695 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:20:55.738 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:20:55.751 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:20:55.778 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:20:55.817 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:20:55.819 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-any. 2024-04-26T01:20:55.846 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T01:20:55.886 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T01:20:56.112 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:20:56.120 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:20:56.128 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:56.130 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package zip. 2024-04-26T01:20:56.146 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T01:20:56.147 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:20:56.190 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:20:56.191 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T01:20:56.191 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:56.349 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:20:56.377 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:20:56.416 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:20:56.467 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package unzip. 2024-04-26T01:20:56.489 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:20:56.494 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T01:20:56.517 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:20:56.525 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:20:56.556 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:20:56.740 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:20:56.743 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:20:56.766 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:56.770 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:20:56.810 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:56.811 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:20:56.843 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:20:56.869 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:20:56.901 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:20:56.904 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package luarocks. 2024-04-26T01:20:56.931 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T01:20:56.970 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:20:57.117 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:20:57.144 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:20:57.163 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:20:57.184 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:20:57.190 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:57.229 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:57.328 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librgw2. 2024-04-26T01:20:57.356 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../025-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:57.386 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:57.487 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:20:57.513 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:57.553 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:57.571 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:20:57.574 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:20:57.598 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:20:57.601 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:20:57.637 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:20:57.640 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:20:57.939 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T01:20:57.941 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:20:57.943 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:20:57.955 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:57.967 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:20:57.971 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:20:57.982 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:20:57.997 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:58.007 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:20:58.008 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:58.009 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:20:58.048 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:58.418 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:20:58.446 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:20:58.463 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:20:58.467 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T01:20:58.485 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:20:58.490 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:20:58.494 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T01:20:58.529 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:58.533 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:20:58.805 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:20:58.831 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:20:58.862 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T01:20:58.879 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:20:58.888 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:58.928 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:59.233 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:20:59.259 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:20:59.273 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-common. 2024-04-26T01:20:59.299 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:20:59.300 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:59.339 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:59.461 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:20:59.488 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:20:59.538 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:20:59.602 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:20:59.629 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:20:59.669 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:20:59.717 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:20:59.734 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:20:59.775 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:20:59.930 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:20:59.956 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:20:59.988 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:21:00.108 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:21:00.134 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:21:00.174 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:21:00.287 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:21:00.315 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:21:00.325 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:21:00.351 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:21:00.378 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:00.394 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:21:00.660 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:21:00.686 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:21:00.686 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:21:00.699 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:21:00.705 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-base. 2024-04-26T01:21:00.712 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:21:00.726 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:21:00.727 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:21:00.732 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:00.752 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:21:00.757 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:00.782 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:01.026 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:21:01.054 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:21:01.055 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:21:01.055 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:21:01.081 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:21:01.082 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:21:01.092 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:01.122 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:21:01.122 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:21:01.310 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T01:21:01.336 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T01:21:01.374 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:21:01.376 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:21:01.400 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:21:01.441 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:21:01.441 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:21:01.457 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:21:01.500 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:21:01.645 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T01:21:01.672 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T01:21:01.712 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:21:01.718 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:21:01.743 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:21:01.745 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:01.770 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:21:01.778 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:21:01.808 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:21:01.810 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:21:01.844 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:21:01.860 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:02.032 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T01:21:02.059 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T01:21:02.098 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:21:02.113 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:21:02.139 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:21:02.147 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:21:02.173 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:21:02.188 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:21:02.214 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:21:02.457 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:21:02.483 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:21:02.515 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:21:02.558 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:21:02.585 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:21:02.625 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:21:02.651 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T01:21:02.678 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T01:21:02.718 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:21:02.802 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:21:02.828 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:21:02.868 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:21:02.979 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T01:21:02.988 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:21:02.989 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:21:03.006 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T01:21:03.014 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:21:03.016 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:21:03.037 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:21:03.054 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:21:03.055 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:03.171 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:21:03.197 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:21:03.237 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:21:03.315 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T01:21:03.341 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:21:03.342 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T01:21:03.349 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:21:03.369 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:21:03.375 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:21:03.381 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T01:21:03.408 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:03.415 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:21:03.591 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:21:03.618 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:21:03.649 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:21:03.685 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-portend. 2024-04-26T01:21:03.686 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:21:03.712 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T01:21:03.714 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:21:03.718 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:21:03.744 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:21:03.745 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:21:03.751 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T01:21:03.785 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:21:03.936 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:21:03.962 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:21:03.989 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:21:04.002 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:21:04.005 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:21:04.038 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:21:04.046 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T01:21:04.073 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T01:21:04.105 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:21:04.112 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T01:21:04.132 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:21:04.174 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:21:04.314 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:21:04.340 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:21:04.348 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T01:21:04.359 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:21:04.364 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T01:21:04.387 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:21:04.397 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:21:04.407 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:21:04.425 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:21:04.483 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:21:04.509 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:21:04.549 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:21:04.700 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:21:04.726 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:21:04.760 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T01:21:04.766 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:21:04.770 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:21:04.789 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T01:21:04.798 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:21:04.835 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T01:21:04.836 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:21:04.852 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:21:04.868 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:04.910 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:05.061 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:21:05.076 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:21:05.105 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T01:21:05.111 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:21:05.129 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T01:21:05.148 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:21:05.163 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T01:21:05.175 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:21:05.214 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:21:05.249 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:21:05.264 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:05.298 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:05.380 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:21:05.407 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:21:05.441 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T01:21:05.447 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:21:05.468 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T01:21:05.508 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:21:05.509 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:21:05.537 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:21:05.567 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:21:05.603 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:21:05.630 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:21:05.670 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:21:05.758 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:21:05.784 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:21:05.819 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-mako. 2024-04-26T01:21:05.824 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:21:05.846 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T01:21:05.886 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:21:05.973 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:21:05.999 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:21:06.039 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:21:06.194 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:21:06.220 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:21:06.222 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:21:06.239 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:06.239 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T01:21:06.264 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:21:06.266 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T01:21:06.272 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:06.305 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:21:06.350 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:21:06.366 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:06.400 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:06.583 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:21:06.584 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T01:21:06.589 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:21:06.610 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T01:21:06.611 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:21:06.615 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:21:06.641 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:21:06.650 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:21:06.655 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:21:06.796 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:21:06.823 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:06.862 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:06.920 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webob. 2024-04-26T01:21:06.924 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:21:06.946 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T01:21:06.947 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:21:06.977 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:21:06.983 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:21:06.986 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:21:07.005 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:07.044 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:21:07.286 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:21:07.296 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package jq. 2024-04-26T01:21:07.301 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:21:07.324 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:07.330 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T01:21:07.344 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:21:07.358 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T01:21:07.400 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:21:07.404 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:21:07.465 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:21:07.492 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:07.532 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:07.615 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package socat. 2024-04-26T01:21:07.622 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:21:07.632 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:21:07.637 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:21:07.672 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:21:07.673 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:07.708 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T01:21:07.735 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T01:21:07.775 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:21:07.958 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:21:07.960 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:21:07.984 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:21:07.988 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:21:08.024 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:21:08.026 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:08.111 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-paste. 2024-04-26T01:21:08.138 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T01:21:08.178 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:21:08.336 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:21:08.347 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:21:08.362 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:21:08.364 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:08.396 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:08.402 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:21:08.506 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T01:21:08.534 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T01:21:08.543 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph. 2024-04-26T01:21:08.556 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:08.572 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:21:08.593 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:08.790 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:21:08.817 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:21:08.857 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:21:08.859 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T01:21:08.886 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T01:21:08.913 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:21:08.925 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:21:08.939 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:08.979 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:09.187 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T01:21:09.193 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:21:09.214 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T01:21:09.219 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:21:09.245 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T01:21:09.259 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:21:09.349 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:21:09.376 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:09.415 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:09.520 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:21:09.536 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:21:09.565 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T01:21:09.570 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:21:09.591 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T01:21:09.631 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:21:09.793 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:21:09.820 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:09.840 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:21:09.856 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:21:09.859 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:09.899 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:21:09.977 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T01:21:10.004 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T01:21:10.044 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T01:21:10.188 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:21:10.193 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:21:10.214 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:21:10.220 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:21:10.254 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:21:10.259 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:21:10.364 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-idna. 2024-04-26T01:21:10.391 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T01:21:10.430 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T01:21:10.538 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:21:10.564 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:10.573 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:21:10.589 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:21:10.604 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:10.623 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:10.659 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:21:10.689 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:10.725 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:10.742 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T01:21:10.769 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T01:21:10.808 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:21:11.037 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:21:11.050 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:21:11.054 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:11.078 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:11.095 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:11.120 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests. 2024-04-26T01:21:11.125 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:11.147 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T01:21:11.186 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:21:11.415 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:21:11.443 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:21:11.456 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T01:21:11.472 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T01:21:11.481 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:11.505 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:21:11.512 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:21:11.539 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:21:11.586 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:21:11.637 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:21:11.664 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:21:11.703 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:11.792 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T01:21:11.808 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:11.811 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:21:11.839 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:21:11.842 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:11.877 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:21:11.915 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:21:11.941 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:21:11.981 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:21:12.014 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:21:12.041 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:21:12.080 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:21:12.225 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T01:21:12.248 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:21:12.249 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:12.276 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:21:12.288 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:12.314 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:12.317 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:21:12.344 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:12.384 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:12.434 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:21:12.460 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:12.500 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:12.600 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T01:21:12.627 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T01:21:12.666 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:21:12.761 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:21:12.770 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:21:12.789 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:21:12.797 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:12.827 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:12.828 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:12.862 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:21:12.889 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:21:12.928 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:21:13.052 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T01:21:13.071 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:21:13.079 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T01:21:13.088 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:21:13.130 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:13.135 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T01:21:13.231 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:21:13.258 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:21:13.297 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:21:13.441 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:21:13.455 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T01:21:13.467 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:13.483 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:13.507 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:13.522 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:13.583 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:21:13.611 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:21:13.617 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:21:13.644 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:13.650 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:13.684 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:13.885 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:21:13.902 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:21:13.917 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T01:21:13.935 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:21:13.944 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:13.983 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:14.238 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:21:14.266 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:21:14.304 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:14.529 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T01:21:14.556 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:14.585 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph. 2024-04-26T01:21:14.587 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:14.609 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:14.615 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:21:14.644 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:21:14.651 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:14.682 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:14.864 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:21:14.879 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:21:14.922 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:21:14.954 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:21:14.981 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:15.019 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:21:15.020 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:15.046 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:21:15.085 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:21:15.407 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:21:15.413 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:21:15.417 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:21:15.434 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:15.441 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:21:15.444 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:21:15.474 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:15.479 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:15.483 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:15.640 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph. 2024-04-26T01:21:15.656 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../066-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:15.698 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:15.854 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:21:15.881 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:21:15.920 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:15.933 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:21:15.962 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:21:15.985 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:21:16.000 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:16.012 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:16.018 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T01:21:16.045 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:16.086 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:16.086 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:16.198 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:21:16.214 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:21:16.247 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:16.294 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:21:16.322 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:21:16.360 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:21:16.404 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:21:16.429 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T01:21:16.431 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:21:16.456 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:16.470 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:21:16.496 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:16.596 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:21:16.624 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:21:16.654 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:21:16.840 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:21:16.856 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:21:16.890 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:16.890 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephadm. 2024-04-26T01:21:16.917 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../069-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:16.939 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:21:16.957 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:16.967 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:16.991 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:21:17.019 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:21:17.066 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:17.108 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:17.268 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T01:21:17.295 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T01:21:17.334 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T01:21:17.343 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:21:17.360 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:21:17.401 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:17.612 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T01:21:17.628 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:21:17.670 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:17.946 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:21:17.972 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:21:18.011 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:18.105 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:21:18.111 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:21:18.134 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:18.138 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:21:18.163 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:18.178 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:18.331 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:21:18.358 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:21:18.398 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:21:18.422 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:21:18.438 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:21:18.480 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:18.700 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-babel. 2024-04-26T01:21:18.728 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T01:21:18.759 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:21:18.767 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:21:18.783 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:18.786 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:18.795 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:21:18.825 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:21:18.826 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:19.075 INFO:teuthology.orchestra.run.smithi045.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:21:19.103 INFO:teuthology.orchestra.run.smithi045.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:19.111 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T01:21:19.128 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:21:19.139 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T01:21:19.141 INFO:teuthology.orchestra.run.smithi045.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:19.154 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:21:19.155 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:21:19.181 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:21:19.186 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:21:19.194 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:21:19.220 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:21:19.445 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:19.506 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T01:21:19.515 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:21:19.531 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:21:19.533 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:19.543 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:21:19.558 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:21:19.573 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:19.573 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:21:19.598 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:21:19.712 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:21:19.838 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:19.860 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:21:19.887 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:21:19.918 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:21:19.926 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:21:19.945 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T01:21:19.953 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:19.984 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:19.984 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T01:21:20.116 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:21:20.221 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:21:20.248 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:21:20.254 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-routes. 2024-04-26T01:21:20.281 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T01:21:20.287 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:21:20.320 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:21:20.500 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:21:20.607 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:21:20.609 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T01:21:20.634 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:20.635 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:21:20.673 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:20.674 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:21:20.939 INFO:teuthology.orchestra.run.smithi045.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:21:21.014 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:21.323 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:21:21.339 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:21:21.354 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:21:21.371 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:21.380 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:21:21.412 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:21.647 INFO:teuthology.orchestra.run.smithi045.stdout:Adding system user cephadm....done 2024-04-26T01:21:21.765 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:21:21.778 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:21:21.792 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:21:21.828 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T01:21:21.832 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:21:21.850 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T01:21:21.876 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:21:21.895 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:21:21.903 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:21:21.942 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:22.054 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:21:22.126 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:21:22.153 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:22.184 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:21:22.229 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:21:22.244 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:21:22.287 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:22.330 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T01:21:22.384 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T01:21:22.423 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:22.437 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package jq. 2024-04-26T01:21:22.453 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:22.457 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:21:22.495 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:21:22.592 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:21:22.619 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:21:22.659 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:22.731 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package socat. 2024-04-26T01:21:22.747 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:21:22.789 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:22.793 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T01:21:22.821 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T01:21:22.826 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:21:22.860 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:23.069 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:21:23.084 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:21:23.107 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:21:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T01:21:23.150 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:23.174 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T01:21:23.213 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:23.334 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:21:23.350 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:23.470 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:23.497 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:21:23.524 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:23.554 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:21:23.563 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:23.873 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:23.971 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T01:21:23.999 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:24.115 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:24.241 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:24.499 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:21:24.526 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:21:24.550 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:21:24.573 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:24.843 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:21:24.943 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:21:24.971 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:21:25.018 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:25.052 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T01:21:25.080 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T01:21:25.118 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:25.130 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:21:25.413 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T01:21:25.440 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T01:21:25.447 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:21:25.474 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:21:25.479 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:25.522 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:21:25.529 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:25.800 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T01:21:25.814 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:25.827 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T01:21:25.842 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:21:25.866 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:21:25.867 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:21:25.869 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:21:25.894 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:25.908 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:21:25.933 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:26.090 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:26.177 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T01:21:26.205 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T01:21:26.229 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:21:26.229 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:21:26.243 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T01:21:26.245 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:26.256 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:21:26.287 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:26.295 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:21:26.358 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:21:26.534 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:26.547 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T01:21:26.575 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T01:21:26.581 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:21:26.609 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:21:26.614 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T01:21:26.648 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:26.685 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:21:26.791 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:21:26.818 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:21:26.836 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:21:26.857 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:21:26.927 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:21:26.934 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T01:21:26.943 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:21:26.961 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T01:21:26.994 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:21:27.000 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:21:27.104 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:21:27.127 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:21:27.154 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:21:27.194 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:21:27.253 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T01:21:27.269 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T01:21:27.303 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T01:21:27.356 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:21:27.384 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:21:27.422 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:27.472 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:21:27.494 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:21:27.522 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:21:27.548 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:21:27.549 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:21:27.588 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:21:27.615 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T01:21:27.643 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T01:21:27.674 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:27.681 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:21:27.836 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:21:27.864 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:21:27.911 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:28.034 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:28.160 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:21:28.245 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:21:28.261 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:28.303 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:28.345 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T01:21:28.364 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:28.403 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:28.428 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:21:28.439 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:21:28.455 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:21:28.488 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:28.598 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:21:28.625 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:21:28.664 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:21:28.729 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:28.850 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:21:28.877 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:21:29.355 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:21:29.355 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T01:21:29.356 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:29.357 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:21:29.357 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:21:29.357 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:21:29.362 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package jq. 2024-04-26T01:21:29.362 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:29.378 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:29.400 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:21:29.412 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:21:29.581 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:21:29.608 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:21:29.631 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package socat. 2024-04-26T01:21:29.647 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:29.648 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:21:29.661 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package jq. 2024-04-26T01:21:29.678 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T01:21:29.698 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:29.711 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T01:21:29.859 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:21:29.917 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:21:29.944 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:21:29.947 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package socat. 2024-04-26T01:21:29.964 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T01:21:29.983 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:29.997 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:30.018 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:21:30.034 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:21:30.076 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:30.126 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:21:30.259 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T01:21:30.260 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:30.287 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T01:21:30.311 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:21:30.317 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:30.338 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:21:30.386 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:21:30.386 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:21:30.439 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:21:30.455 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:30.497 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:30.656 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:21:30.663 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-test. 2024-04-26T01:21:30.671 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:21:30.679 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:30.687 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:21:30.705 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:30.729 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:31.105 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:21:31.118 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:21:31.145 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:21:31.184 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:31.356 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:21:31.479 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:21:31.506 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:21:31.545 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:21:31.741 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:21:31.856 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:21:31.884 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:21:31.923 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:21:32.043 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:21:32.218 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:21:32.245 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:21:32.285 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:32.402 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:21:32.537 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:21:32.553 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:21:32.595 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:32.653 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:32.780 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:32.861 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:21:32.888 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:32.935 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:33.001 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T01:21:33.029 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-26T01:21:33.067 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:33.081 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:21:33.291 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:21:33.306 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:21:33.318 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:33.333 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:33.349 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:33.379 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T01:21:33.381 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:33.390 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:33.396 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:33.429 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:33.709 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:21:33.716 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T01:21:33.736 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:21:33.743 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T01:21:33.775 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:33.782 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:34.106 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:21:34.120 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T01:21:34.133 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:21:34.135 INFO:teuthology.orchestra.run.smithi181.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:21:34.148 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T01:21:34.163 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:34.186 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:21:34.197 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:21:34.201 INFO:teuthology.orchestra.run.smithi181.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:34.514 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:34.557 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:21:34.585 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:21:34.596 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:34.601 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T01:21:34.629 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T01:21:34.649 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:34.668 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:34.765 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:21:34.848 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:34.891 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:34.982 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:21:35.071 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:21:35.082 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package pkg-config. 2024-04-26T01:21:35.099 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:21:35.110 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T01:21:35.137 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:35.140 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:35.154 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:21:35.258 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:21:35.384 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:35.385 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:21:35.399 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T01:21:35.401 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:21:35.427 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T01:21:35.443 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:35.465 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:35.486 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:21:35.510 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:21:35.811 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:35.872 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:21:35.885 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T01:21:35.900 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:21:35.913 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T01:21:35.938 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:35.939 INFO:teuthology.orchestra.run.smithi181.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:21:35.951 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:36.009 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:36.071 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:21:36.241 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:21:36.254 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T01:21:36.269 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:21:36.282 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T01:21:36.299 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:21:36.320 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T01:21:36.330 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:36.456 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:36.525 INFO:teuthology.orchestra.run.smithi181.stdout:Adding system user cephadm....done 2024-04-26T01:21:36.587 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:21:36.614 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:21:36.649 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T01:21:36.653 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:36.657 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:21:36.677 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T01:21:36.707 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:36.749 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:21:36.948 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:21:36.949 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:21:36.976 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:21:37.000 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:21:37.002 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T01:21:37.023 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:37.030 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T01:21:37.068 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:37.209 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:37.268 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:21:37.321 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:21:37.335 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:21:37.338 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:21:37.371 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:21:37.401 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-py. 2024-04-26T01:21:37.429 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T01:21:37.468 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T01:21:37.527 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:21:37.595 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:21:37.693 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:21:37.721 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:21:37.759 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:37.779 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T01:21:37.795 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:21:37.807 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T01:21:37.846 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:37.854 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:21:37.989 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:21:38.071 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:21:38.197 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:21:38.263 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:21:38.292 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:21:38.298 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:38.325 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T01:21:38.329 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:38.353 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T01:21:38.392 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:38.473 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:21:38.591 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:21:38.645 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:21:38.649 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-toml. 2024-04-26T01:21:38.659 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:38.673 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:21:38.678 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T01:21:38.703 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:21:38.715 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T01:21:38.884 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:21:38.910 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:21:39.002 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:21:39.006 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T01:21:39.030 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:21:39.034 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T01:21:39.068 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:21:39.072 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:21:39.185 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:21:39.195 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:21:39.376 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:21:39.404 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:21:39.405 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T01:21:39.434 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T01:21:39.434 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:39.436 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:21:39.471 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:39.488 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:21:39.596 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:21:39.704 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:21:39.720 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:21:39.721 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:39.730 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:39.732 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T01:21:39.749 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T01:21:39.762 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:39.790 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:39.811 INFO:teuthology.orchestra.run.smithi045.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:21:39.998 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:40.040 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:21:40.166 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:21:40.289 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:40.503 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:21:40.531 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:40.533 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package radosgw. 2024-04-26T01:21:40.559 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:21:40.562 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../116-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:40.569 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:40.600 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:40.600 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:21:40.726 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:40.827 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:21:40.877 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:21:40.953 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:21:41.011 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:21:41.212 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:41.288 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:21:41.338 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:21:41.338 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:21:41.355 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:41.397 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:41.453 INFO:teuthology.orchestra.run.smithi071.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T01:21:41.469 INFO:teuthology.orchestra.run.smithi071.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-26T01:21:41.519 INFO:teuthology.orchestra.run.smithi071.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:41.647 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:21:41.702 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:41.728 INFO:teuthology.orchestra.run.smithi181.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:21:41.760 INFO:teuthology.orchestra.run.smithi181.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:21:41.795 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T01:21:41.816 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:21:41.933 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:41.940 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:21:41.985 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:21:42.112 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:42.208 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:21:42.234 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T01:21:42.293 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:42.334 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:21:42.359 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T01:21:42.386 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:21:42.418 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:21:42.610 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:21:42.624 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T01:21:42.695 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:21:42.716 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:21:42.886 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:21:42.921 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T01:21:42.954 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:43.125 INFO:teuthology.orchestra.run.smithi037.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:21:43.188 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:43.239 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:21:43.239 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:21:43.322 INFO:teuthology.orchestra.run.smithi071.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T01:21:43.350 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:43.395 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:43.608 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:21:43.655 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user cephadm....done 2024-04-26T01:21:43.811 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:21:43.886 INFO:teuthology.orchestra.run.smithi071.stdout:Adding system user cephadm....done 2024-04-26T01:21:43.918 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:21:43.975 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:21:44.001 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T01:21:44.217 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:21:44.226 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:21:44.254 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:21:44.278 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T01:21:44.352 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:44.380 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:44.486 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:21:44.506 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:44.521 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:44.597 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T01:21:44.615 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:21:44.622 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:21:44.723 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T01:21:44.813 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:21:44.882 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:21:44.927 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:44.966 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T01:21:45.053 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:45.133 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:21:45.179 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:21:45.180 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:21:45.217 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T01:21:45.259 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:21:45.343 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T01:21:45.423 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:21:45.463 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:45.565 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:45.602 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:45.674 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:45.678 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T01:21:45.791 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:21:45.894 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:21:45.894 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:21:45.963 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:46.080 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T01:21:46.084 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:21:46.206 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:21:46.339 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T01:21:46.392 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:46.427 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:21:46.474 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:21:46.526 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:46.599 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T01:21:46.652 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:46.687 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:46.726 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:21:46.778 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:21:46.813 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:46.850 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T01:21:46.978 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:47.080 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:21:47.110 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T01:21:47.236 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:47.377 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T01:21:47.390 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:47.488 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:47.747 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:21:47.753 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:47.884 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:48.013 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T01:21:48.065 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:48.124 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:21:48.138 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T01:21:48.191 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:48.248 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T01:21:48.250 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:21:48.351 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:48.357 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T01:21:48.476 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:48.518 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:21:48.580 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:48.602 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:48.608 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T01:21:48.799 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:48.901 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:21:48.933 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:21:48.933 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:21:48.988 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:21:49.004 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T01:21:49.049 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T01:21:49.106 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:49.111 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T01:21:49.162 INFO:teuthology.orchestra.run.smithi045.stdout:Adding group ceph....done 2024-04-26T01:21:49.192 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:21:49.237 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T01:21:49.318 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:49.461 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:21:49.474 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:49.548 INFO:teuthology.orchestra.run.smithi045.stdout:Adding system user ceph....done 2024-04-26T01:21:49.600 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:21:49.605 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:49.707 INFO:teuthology.orchestra.run.smithi045.stdout:Setting system user ceph properties....done 2024-04-26T01:21:49.714 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T01:21:49.726 INFO:teuthology.orchestra.run.smithi045.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:21:49.771 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:49.870 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:21:49.884 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:21:49.983 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T01:21:50.005 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:21:50.152 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:50.205 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:21:50.240 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:50.242 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T01:21:50.365 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:50.454 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:21:50.543 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T01:21:50.554 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:50.658 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:21:50.669 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T01:21:50.695 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:50.821 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:50.910 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:21:51.177 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:21:51.198 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:21:51.326 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:21:51.326 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:21:51.363 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T01:21:51.437 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:21:51.466 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:21:51.592 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:51.630 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T01:21:51.696 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:21:51.719 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:21:51.748 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T01:21:51.815 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:51.866 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T01:21:51.947 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:21:52.003 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:21:52.073 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:21:52.158 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T01:21:52.177 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:21:52.366 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:21:52.371 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:21:52.492 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:21:52.576 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T01:21:52.613 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:21:52.659 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:52.785 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:21:52.827 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T01:21:52.879 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:21:52.880 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:21:52.999 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:21:53.086 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:21:53.212 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T01:21:53.292 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:21:53.345 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:21:53.386 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:53.454 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:21:53.522 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T01:21:53.564 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:21:53.614 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:21:53.614 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:21:53.635 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:21:53.653 INFO:teuthology.orchestra.run.smithi181.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:21:53.886 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:53.890 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T01:21:53.899 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:21:53.987 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:54.025 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:21:54.141 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T01:21:54.162 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:54.259 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:54.263 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:21:54.418 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:21:54.462 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:21:54.462 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:21:54.568 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T01:21:54.573 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:54.677 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:21:54.806 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:21:54.886 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T01:21:54.980 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:55.054 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:55.090 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:55.155 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:21:55.224 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:55.443 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:21:55.444 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:21:55.448 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:21:55.688 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:55.741 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:21:55.939 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:55.959 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:56.017 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:21:56.074 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:21:56.110 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:56.160 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:21:56.184 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T01:21:56.236 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:56.342 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:21:56.387 INFO:teuthology.orchestra.run.smithi045.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:56.426 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:56.435 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:21:56.468 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:56.561 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T01:21:56.594 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:21:56.703 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:21:56.739 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:21:56.828 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T01:21:56.832 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:21:56.879 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:56.954 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T01:21:57.009 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:21:57.031 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:21:57.080 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T01:21:57.113 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:21:57.381 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:57.382 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T01:21:57.408 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:21:57.507 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:57.658 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T01:21:57.693 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:21:57.809 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:21:57.909 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:21:58.018 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T01:21:58.029 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:21:58.093 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:21:58.155 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:58.168 INFO:teuthology.orchestra.run.smithi045.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:21:58.281 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:58.320 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T01:21:58.336 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:21:58.407 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:21:58.571 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:21:58.579 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T01:21:58.735 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:58.838 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:21:58.846 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T01:21:58.856 INFO:teuthology.orchestra.run.smithi045.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:21:58.862 DEBUG:teuthology.parallel:result is None 2024-04-26T01:21:58.863 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:59.106 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T01:21:59.107 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:21:59.232 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:21:59.320 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:21:59.349 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T01:21:59.508 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:21:59.600 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T01:21:59.622 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:59.634 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:21:59.743 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T01:21:59.748 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:59.874 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:21:59.919 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:22:00.019 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T01:22:00.114 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:22:00.114 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:22:00.145 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T01:22:00.195 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:22:00.438 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:22:00.446 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T01:22:00.564 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:22:00.600 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:00.690 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:22:00.726 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:00.755 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T01:22:00.780 INFO:teuthology.orchestra.run.smithi071.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:22:00.852 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:00.978 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:22:01.007 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T01:22:01.025 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:22:01.141 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T01:22:01.143 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:22:01.259 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T01:22:01.330 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T01:22:01.503 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:22:01.585 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T01:22:01.711 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T01:22:01.746 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:22:01.881 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:22:02.104 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T01:22:02.165 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:22:02.199 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:02.291 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:22:02.334 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:02.372 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T01:22:02.434 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:02.498 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T01:22:02.560 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:02.617 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:22:02.686 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:02.758 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T01:22:02.875 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T01:22:02.902 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:22:03.153 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:22:03.185 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T01:22:03.229 INFO:teuthology.orchestra.run.smithi181.stdout:Adding group ceph....done 2024-04-26T01:22:03.271 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:22:03.503 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T01:22:03.547 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:22:03.556 INFO:teuthology.orchestra.run.smithi181.stdout:Adding system user ceph....done 2024-04-26T01:22:03.699 INFO:teuthology.orchestra.run.smithi181.stdout:Setting system user ceph properties....done 2024-04-26T01:22:03.716 INFO:teuthology.orchestra.run.smithi181.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:22:03.795 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T01:22:03.823 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:22:03.852 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:22:03.921 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T01:22:04.169 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:22:04.190 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:22:04.197 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T01:22:04.465 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T01:22:04.529 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:22:04.730 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:04.810 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T01:22:04.814 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:22:04.924 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:05.150 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:22:05.212 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T01:22:05.276 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:05.377 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:05.479 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:22:05.505 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T01:22:05.514 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:22:05.514 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:22:05.809 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:05.858 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T01:22:05.935 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:05.967 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:05.992 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:06.056 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:22:06.068 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:06.177 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T01:22:06.321 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:06.329 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:22:06.447 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:06.513 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:06.573 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:06.639 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:06.765 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T01:22:06.785 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:22:06.785 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:22:06.794 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.015 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:22:07.015 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:22:07.059 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.160 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.250 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.261 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.376 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.481 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:22:07.482 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T01:22:07.485 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.487 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:07.611 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:22:07.702 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:22:07.703 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:22:08.004 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:08.130 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:08.155 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:08.247 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:08.357 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T01:22:08.435 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:22:08.436 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:22:08.757 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:08.883 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:08.940 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.009 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.066 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.110 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.200 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.211 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.390 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:22:09.391 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:22:09.611 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.737 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.746 INFO:teuthology.orchestra.run.smithi071.stdout:Adding group ceph....done 2024-04-26T01:22:09.852 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.862 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.988 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:09.995 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:10.064 INFO:teuthology.orchestra.run.smithi071.stdout:Adding system user ceph....done 2024-04-26T01:22:10.114 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:10.129 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:10.234 INFO:teuthology.orchestra.run.smithi071.stdout:Setting system user ceph properties....done 2024-04-26T01:22:10.249 INFO:teuthology.orchestra.run.smithi071.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:22:10.280 INFO:teuthology.orchestra.run.smithi181.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:10.397 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:22:10.642 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:22:10.657 INFO:teuthology.orchestra.run.smithi037.stdout:Adding group ceph....done 2024-04-26T01:22:10.697 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:22:10.742 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:22:10.886 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:22:11.034 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user ceph....done 2024-04-26T01:22:11.138 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:11.202 INFO:teuthology.orchestra.run.smithi037.stdout:Setting system user ceph properties....done 2024-04-26T01:22:11.214 INFO:teuthology.orchestra.run.smithi037.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T01:22:11.281 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:11.357 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T01:22:11.644 INFO:teuthology.orchestra.run.smithi181.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:22:11.687 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T01:22:11.752 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:22:11.752 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:22:12.182 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:12.192 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:12.273 INFO:teuthology.orchestra.run.smithi181.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:22:12.278 DEBUG:teuthology.parallel:result is None 2024-04-26T01:22:12.467 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:12.536 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:22:12.925 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:22:12.925 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T01:22:13.002 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:13.210 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:22:13.210 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:22:13.402 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:13.662 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:13.775 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T01:22:13.889 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:22:13.890 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:22:14.245 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:14.347 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:14.483 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:22:14.483 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T01:22:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:22:14.633 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:22:15.005 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:15.099 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:15.225 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:15.359 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:15.455 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:22:15.456 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T01:22:15.561 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:22:15.561 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:22:15.932 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:16.027 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:16.153 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:16.215 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:22:16.215 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T01:22:16.279 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:16.430 INFO:teuthology.orchestra.run.smithi071.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:16.694 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:16.792 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:22:16.818 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:16.893 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:22:16.969 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:17.036 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:22:17.194 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:22:17.194 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T01:22:17.803 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:18.013 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:18.139 INFO:teuthology.orchestra.run.smithi071.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:22:18.156 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:18.307 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-26T01:22:18.676 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T01:22:18.777 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T01:22:18.842 INFO:teuthology.orchestra.run.smithi071.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:22:18.847 DEBUG:teuthology.parallel:result is None 2024-04-26T01:22:18.929 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T01:22:20.022 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T01:22:20.676 INFO:teuthology.orchestra.run.smithi037.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T01:22:20.680 DEBUG:teuthology.parallel:result is None 2024-04-26T01:22:20.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:22:20.831 DEBUG:teuthology.orchestra.run.smithi037:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:22:20.850 INFO:teuthology.orchestra.run.smithi037.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:20.851 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:20.851 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:22:20.853 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:22:20.996 DEBUG:teuthology.orchestra.run.smithi045:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:22:21.016 INFO:teuthology.orchestra.run.smithi045.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:21.016 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:21.016 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:22:21.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:22:21.155 DEBUG:teuthology.orchestra.run.smithi071:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:22:21.174 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:21.174 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:21.175 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:22:21.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-26T01:22:21.443 DEBUG:teuthology.orchestra.run.smithi181:> dpkg-query -W -f '${Version}' ceph 2024-04-26T01:22:21.463 INFO:teuthology.orchestra.run.smithi181.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:21.463 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-26T01:22:21.463 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-26T01:22:21.465 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T01:22:21.465 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:21.465 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:22:21.480 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:22:21.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:22:21.494 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:22:21.494 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:22:21.508 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:22:21.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T01:22:21.523 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T01:22:21.523 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:21.523 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:22:21.537 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:22:21.592 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:22:21.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:22:21.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:22:21.661 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:22:21.661 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:22:21.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:22:21.730 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:22:21.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T01:22:21.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T01:22:21.797 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T01:22:21.797 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:21.797 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:22:21.812 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:22:21.868 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:22:21.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:22:21.882 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:22:21.936 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:22:21.937 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:22:21.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:22:22.002 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:22:22.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T01:22:22.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T01:22:22.069 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T01:22:22.069 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:22.070 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:22:22.083 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:22:22.140 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:22:22.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:22:22.154 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:22:22.209 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:22:22.209 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:22:22.222 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:22:22.275 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:22:22.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T01:22:22.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T01:22:22.342 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T01:22:22.435 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T01:22:22.435 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-04-26T01:22:22.438 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /var/log/ceph 2024-04-26T01:22:22.440 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-04-26T01:22:22.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-04-26T01:22:22.455 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T01:22:22.455 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:22:22.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:22:22.495 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:22:22.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T01:22:22.509 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T01:22:22.509 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:22:22.549 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:22:22.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:22:22.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T01:22:22.582 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T01:22:22.583 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-26T01:22:22.583 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000', 'branch': 'squid', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7674000', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 9886, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '30270/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b22e2ebdeb24376882b7bda2a7329c8cccc2276a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a', 'targets': {'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGcxhBjkX8tb5Wy+F24m7wrrFVlxtgpld3aT4x0qj8ih5k4DFrcx/2f3mQfwLvaBLPzmr4upe0vzs1AXF9NoLSY=', 'smithi045.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN568PHByclQZBg9C/JgU/9kWxRTDJyKuOAnv56HtX4obqVy69BIZru/AWFJlpgZy5iA6/0FrdSrJISHEySIERA=', 'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFamGPdQPSnJICWOC9Wbj/ORtCpsQNWbZXU5J4LP6zXmQ8y1hvC1fgtmK7R3NKznhYFsRsap6YXnXDXX4QuBRRo=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLOab7Xe2nDHX4inIWcwqEaS+VU9btv4yUIgQcapfkkr307sEivv3ZNIFwOI6dPgw1AUzSFVCH6WmAQXxXCRrxQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_21:00:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T01:22:22.583 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:22:22.605 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:22:22.610 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:22:22.612 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T01:22:22.618 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:22:22.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:22:22.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:22:22.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T01:22:22.675 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:22.675 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:22:22.712 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:22:22.712 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-04-26T01:22:22.760 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:22:22.761 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 870 Links: 1 2024-04-26T01:22:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 01:16:31.261602682 +0000 2024-04-26T01:22:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 01:16:30.973605242 +0000 2024-04-26T01:22:22.761 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 01:16:30.973605242 +0000 2024-04-26T01:22:22.761 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T01:22:22.761 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:22:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T01:22:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T01:22:22.815 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000446594 s, 1.1 MB/s 2024-04-26T01:22:22.816 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:22:22.867 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-04-26T01:22:22.916 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:22:22.917 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-04-26T01:22:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 01:16:31.789597988 +0000 2024-04-26T01:22:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 01:16:31.501600548 +0000 2024-04-26T01:22:22.917 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 01:16:31.501600548 +0000 2024-04-26T01:22:22.917 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T01:22:22.917 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:22:22.971 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T01:22:22.971 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T01:22:22.971 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000430374 s, 1.2 MB/s 2024-04-26T01:22:22.972 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:22:23.022 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-04-26T01:22:23.068 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:22:23.068 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:23.068 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-04-26T01:22:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 01:16:32.269593722 +0000 2024-04-26T01:22:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 01:16:32.025595890 +0000 2024-04-26T01:22:23.069 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 01:16:32.025595890 +0000 2024-04-26T01:22:23.069 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T01:22:23.069 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:22:23.123 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T01:22:23.123 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T01:22:23.123 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000415397 s, 1.2 MB/s 2024-04-26T01:22:23.124 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:22:23.173 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-04-26T01:22:23.220 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:22:23.220 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 928 Links: 1 2024-04-26T01:22:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-26 01:16:32.797589028 +0000 2024-04-26T01:22:23.220 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-26 01:16:32.525591445 +0000 2024-04-26T01:22:23.221 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-26 01:16:32.525591445 +0000 2024-04-26T01:22:23.221 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-26T01:22:23.221 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:22:23.275 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-26T01:22:23.275 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-26T01:22:23.275 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000393281 s, 1.3 MB/s 2024-04-26T01:22:23.276 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:22:23.326 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:23.326 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:22:23.326 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:22:23.333 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:22:23.333 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-04-26T01:22:23.382 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:22:23.382 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:23.382 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 872 Links: 1 2024-04-26T01:22:23.382 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:23.382 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 01:16:31.638644132 +0000 2024-04-26T01:22:23.382 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 01:16:31.358646620 +0000 2024-04-26T01:22:23.382 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 01:16:31.358646620 +0000 2024-04-26T01:22:23.383 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T01:22:23.383 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:22:23.436 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T01:22:23.436 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T01:22:23.436 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000303295 s, 1.7 MB/s 2024-04-26T01:22:23.437 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:22:23.487 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 01:16:32.134639726 +0000 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 01:16:31.866642107 +0000 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 01:16:31.866642107 +0000 2024-04-26T01:22:23.534 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T01:22:23.535 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:22:23.587 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T01:22:23.588 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T01:22:23.588 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000294098 s, 1.7 MB/s 2024-04-26T01:22:23.589 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:22:23.638 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-04-26T01:22:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:22:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:23.686 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-26T01:22:23.686 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:23.686 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 01:16:32.606635532 +0000 2024-04-26T01:22:23.686 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 01:16:32.346637842 +0000 2024-04-26T01:22:23.686 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 01:16:32.346637842 +0000 2024-04-26T01:22:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T01:22:23.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:22:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T01:22:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T01:22:23.740 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000292411 s, 1.8 MB/s 2024-04-26T01:22:23.741 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:22:23.790 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-04-26 01:16:33.090631233 +0000 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-04-26 01:16:32.846633401 +0000 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-04-26 01:16:32.846633401 +0000 2024-04-26T01:22:23.838 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: - 2024-04-26T01:22:23.839 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:22:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-04-26T01:22:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-04-26T01:22:23.892 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000306625 s, 1.7 MB/s 2024-04-26T01:22:23.893 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:22:23.943 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:23.943 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:22:23.943 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:22:23.949 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:22:23.950 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-04-26T01:22:23.996 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:22:23.996 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:23.996 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-04-26T01:22:23.996 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:23.996 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-26 01:16:28.001073595 +0000 2024-04-26T01:22:23.996 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-26 01:16:27.761101903 +0000 2024-04-26T01:22:23.996 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-26 01:16:27.761101903 +0000 2024-04-26T01:22:23.997 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-26T01:22:23.997 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:22:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-26T01:22:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-26T01:22:24.050 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00044634 s, 1.1 MB/s 2024-04-26T01:22:24.052 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:22:24.105 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-04-26T01:22:24.151 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:22:24.152 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:24.152 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-04-26T01:22:24.152 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:24.152 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-26 01:16:28.497016588 +0000 2024-04-26T01:22:24.152 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-26 01:16:28.245045552 +0000 2024-04-26T01:22:24.152 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-26 01:16:28.245045552 +0000 2024-04-26T01:22:24.152 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-26T01:22:24.152 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:22:24.206 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-26T01:22:24.206 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-26T01:22:24.206 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00041367 s, 1.2 MB/s 2024-04-26T01:22:24.207 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:22:24.256 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 904 Links: 1 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-26 01:16:29.028955551 +0000 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-26 01:16:28.752987166 +0000 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-26 01:16:28.752987166 +0000 2024-04-26T01:22:24.304 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-26T01:22:24.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:22:24.357 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-26T01:22:24.357 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-26T01:22:24.357 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000311344 s, 1.6 MB/s 2024-04-26T01:22:24.359 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:22:24.408 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-26 01:16:29.524900529 +0000 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-26 01:16:29.268928928 +0000 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-26 01:16:29.268928928 +0000 2024-04-26T01:22:24.456 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: - 2024-04-26T01:22:24.456 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:22:24.509 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-26T01:22:24.509 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-26T01:22:24.510 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00035039 s, 1.5 MB/s 2024-04-26T01:22:24.511 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:22:24.561 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:24.561 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:22:24.561 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T01:22:24.567 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T01:22:24.567 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-04-26T01:22:24.615 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T01:22:24.615 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:24.616 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-04-26T01:22:24.616 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:24.616 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-26 01:16:30.057229979 +0000 2024-04-26T01:22:24.616 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-26 01:16:29.817232119 +0000 2024-04-26T01:22:24.616 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-26 01:16:29.817232119 +0000 2024-04-26T01:22:24.616 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-26T01:22:24.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T01:22:24.670 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-26T01:22:24.670 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-26T01:22:24.670 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000439698 s, 1.2 MB/s 2024-04-26T01:22:24.671 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T01:22:24.720 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-26 01:16:30.581225308 +0000 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-26 01:16:30.301227804 +0000 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-26 01:16:30.301227804 +0000 2024-04-26T01:22:24.767 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-26T01:22:24.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T01:22:24.822 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-26T01:22:24.822 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-26T01:22:24.822 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000439193 s, 1.2 MB/s 2024-04-26T01:22:24.823 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T01:22:24.872 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-26 01:16:31.109220601 +0000 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-26 01:16:30.813223239 +0000 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-26 01:16:30.813223239 +0000 2024-04-26T01:22:24.923 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-26T01:22:24.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T01:22:24.977 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-26T01:22:24.977 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-26T01:22:24.977 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000402004 s, 1.3 MB/s 2024-04-26T01:22:24.978 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T01:22:25.028 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-04-26T01:22:25.075 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T01:22:25.075 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T01:22:25.075 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 940 Links: 1 2024-04-26T01:22:25.075 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T01:22:25.075 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-04-26 01:16:31.641215857 +0000 2024-04-26T01:22:25.075 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-04-26 01:16:31.377218211 +0000 2024-04-26T01:22:25.075 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-04-26 01:16:31.377218211 +0000 2024-04-26T01:22:25.076 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2024-04-26T01:22:25.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T01:22:25.129 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-04-26T01:22:25.130 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-04-26T01:22:25.130 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000402964 s, 1.3 MB/s 2024-04-26T01:22:25.131 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T01:22:25.180 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:25.180 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-04-26T01:22:25.180 INFO:tasks.ceph:Generating config... 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T01:22:25.182 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-04-26T01:22:25.183 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-04-26T01:22:25.184 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-26T01:22:25.184 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-04-26T01:22:25.184 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2024-04-26T01:22:25.184 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2024-04-26T01:22:25.184 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:22:25.184 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T01:22:25.184 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T01:22:25.185 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T01:22:25.185 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T01:22:25.185 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T01:22:25.185 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-26T01:22:25.185 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T01:22:25.185 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T01:22:25.221 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T01:22:25.229 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T01:22:25.308 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:22:25.364 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.37'), ('mon.b', '172.21.15.45'), ('mon.c', '172.21.15.71')] 2024-04-26T01:22:25.364 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.37,172.21.15.45,172.21.15.71', 'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-04-26T01:22:25.365 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:25.365 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T01:22:25.412 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.37 --add b 172.21.15.45 --add c 172.21.15.71 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: generated fsid ec0903d9-9475-4d3a-af50-1dbb367f53fb 2024-04-26T01:22:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = quincy 2024-04-26T01:22:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 0 2024-04-26T01:22:25.500 INFO:teuthology.orchestra.run.smithi037.stdout:fsid ec0903d9-9475-4d3a-af50-1dbb367f53fb 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:last_changed 2024-04-26T01:22:25.505042+0000 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-04-26T01:22:25.505042+0000 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:min_mon_release 17 (quincy) 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:election_strategy: 3 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.a 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:1: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.b 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.c 2024-04-26T01:22:25.501 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T01:22:25.502 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T01:22:25.507 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ec0903d9-9475-4d3a-af50-1dbb367f53fb... 2024-04-26T01:22:25.508 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:22:25.553 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:22:25.555 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:22:25.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T01:22:25.581 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-04-26T01:22:25.581 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-04-26T01:22:25.581 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:22:25.581 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-04-26T01:22:25.581 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.581 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.582 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:22:25.583 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-04-26T01:22:25.584 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "172.21.15.37,172.21.15.45,172.21.15.71" 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: mon election default strategy = 3 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = ec0903d9-9475-4d3a-af50-1dbb367f53fb 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 120 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:22:25.585 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug trim objects = true 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:22:25.586 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore compression algorithm = snappy 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore compression mode = aggressive 2024-04-26T01:22:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore zero block detection = True 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 20 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 20 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug reject backfill probability = 0.3 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify cached snaps = True 2024-04-26T01:22:25.588 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify missing on start = True 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd max backfills = 3 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd max markdown count = 1000 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock override recovery settings = True 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue = debug_random 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue cut off = debug_random 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub min interval = 60 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: osd shutdown pgref assert = True 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-04-26T01:22:25.589 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-04-26T01:22:25.590 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.591 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub interval = 300 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-04-26T01:22:25.592 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-04-26T01:22:25.593 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-04-26T01:22:25.593 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-04-26T01:22:25.594 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-04-26T01:22:25.595 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-04-26T01:22:25.596 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.37,172.21.15.45,172.21.15.71" 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: mon election default strategy = 3 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = ec0903d9-9475-4d3a-af50-1dbb367f53fb 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 120 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:22:25.597 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.598 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression algorithm = snappy 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression mode = aggressive 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-04-26T01:22:25.599 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore zero block detection = True 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug reject backfill probability = 0.3 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify cached snaps = True 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify missing on start = True 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: osd max backfills = 3 2024-04-26T01:22:25.600 INFO:teuthology.orchestra.run.smithi181.stdout: osd max markdown count = 1000 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock override recovery settings = True 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue = debug_random 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue cut off = debug_random 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub min interval = 60 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: osd shutdown pgref assert = True 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.601 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.602 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:22:25.603 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub interval = 300 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-04-26T01:22:25.604 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-04-26T01:22:25.605 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:22:25.606 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.607 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "172.21.15.37,172.21.15.45,172.21.15.71" 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 3 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = ec0903d9-9475-4d3a-af50-1dbb367f53fb 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-04-26T01:22:25.608 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 120 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-04-26T01:22:25.609 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.610 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression algorithm = snappy 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:22:25.611 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug reject backfill probability = 0.3 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd max backfills = 3 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd max markdown count = 1000 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub min interval = 60 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: osd shutdown pgref assert = True 2024-04-26T01:22:25.612 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-04-26T01:22:25.613 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.614 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:22:25.615 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:22:25.616 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-04-26T01:22:25.616 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-04-26T01:22:25.616 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: chdir = "" 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: auth supported = cephx 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: filestore xattr use omap = true 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: mon clock drift allowed = 1.000 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: osd crush chooseleaf type = 0 2024-04-26T01:22:25.617 INFO:teuthology.orchestra.run.smithi045.stdout: auth debug = true 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on old message = true 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: ms die on bug = true 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: mon pg warn max object skew = 0 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default size = 2 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.618 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow primary affinity = true 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd allow pg remap = true 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on legacy crush tunables = false 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on crush straw calc version zero = false 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on no sortbitwise = false 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on osd down out interval zero = false 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon warn on too few osds = false 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: mon_allow_pool_size_one = true 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: osd default data pool replay window = 5 2024-04-26T01:22:25.619 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: mon allow pool delete = true 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: mon cluster log file level = debug 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: debug asserts on shutdown = true 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: mon health detail to clog = false 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: mon host = "172.21.15.37,172.21.15.45,172.21.15.71" 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: mon election default strategy = 3 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug inject dispatch delay duration = 0.1 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug inject dispatch delay probability = 0.1 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = ec0903d9-9475-4d3a-af50-1dbb367f53fb 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.620 INFO:teuthology.orchestra.run.smithi045.stdout:[osd] 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd journal size = 100 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub load threshold = 5.0 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub max interval = 120 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock profile = high_recovery_ops 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd recover clone overlap = true 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd recovery max chunk = 1048576 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug shutdown = true 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug op order = true 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify stray on activate = true 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug trim objects = true 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd open classes on start = true 2024-04-26T01:22:25.621 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug pg log writeout = true 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: osd deep scrub update digest min age = 30 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: osd map max advance = 10 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: journal zero on create = true 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: filestore ondisk finisher threads = 3 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: filestore apply finisher threads = 3 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: bdev debug aio = true 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug misdirected ops = true 2024-04-26T01:22:25.622 INFO:teuthology.orchestra.run.smithi045.stdout: debug monc = 20 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore block size = 96636764160 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression algorithm = snappy 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore compression mode = aggressive 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore fsck on mount = True 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: bluestore zero block detection = True 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluefs = 20 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: debug bluestore = 20 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: debug osd = 20 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: debug rocksdb = 10 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd full ratio = 0.9 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd nearfull ratio = 0.8 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: osd blocked scrub grace period = 3600 2024-04-26T01:22:25.623 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug reject backfill probability = 0.3 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify cached snaps = True 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd debug verify missing on start = True 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd failsafe full ratio = 0.95 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd max backfills = 3 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd max markdown count = 1000 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd mclock override recovery settings = True 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd objectstore = bluestore 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue = debug_random 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd op queue cut off = debug_random 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd scrub min interval = 60 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: osd shutdown pgref assert = True 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr] 2024-04-26T01:22:25.624 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: debug mgr = 20 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: mgr/telemetry/nag = false 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout:[mon] 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: debug ms = 1 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: debug mon = 20 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: debug paxos = 20 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: debug auth = 20 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: mon data avail warn = 5 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: mon mgr mkfs grace = 240 2024-04-26T01:22:25.625 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min pgs per osd = 4 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd reporter subtree level = osd 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: mon osd prime pg temp = true 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: mon reweight min bytes per osd = 10 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: # 1m isn't quite enough 2024-04-26T01:22:25.626 INFO:teuthology.orchestra.run.smithi045.stdout: mon_down_mkfs_grace = 2m 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: mon_warn_on_filestore_osds = false 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: mon scrub interval = 300 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout:[client] 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: rgw cache enabled = true 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable ops log = true 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: rgw enable usage log = true 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.a] 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.b] 2024-04-26T01:22:25.627 INFO:teuthology.orchestra.run.smithi045.stdout:[mon.c] 2024-04-26T01:22:25.628 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T01:22:25.629 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T01:22:25.704 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T01:22:25.704 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:25.704 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T01:22:25.710 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:25.710 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T01:22:25.756 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:22:25.756 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:25.756 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:22:25.756 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:22:25.821 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:22:25.822 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:25.868 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:22:25.868 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:22:25.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:22:25.868 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:22:25.894 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:22:25.894 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:25.941 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:22:25.941 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:22:25.941 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:22:25.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:22:25.964 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:22:25.964 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:26.012 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:22:26.012 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:22:26.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T01:22:26.012 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T01:22:26.035 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:22:26.035 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:26.083 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T01:22:26.083 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T01:22:26.083 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T01:22:26.130 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T01:22:26.133 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T01:22:26.176 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T01:22:26.196 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T01:22:26.196 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T01:22:26.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T01:22:26.271 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-26T01:22:26.281 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T01:22:26.282 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi045.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-04-26T01:22:26.282 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T01:22:26.296 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:26.296 INFO:tasks.ceph:role: osd.0 2024-04-26T01:22:26.296 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:22:26.296 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:22:26.362 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:26.362 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:26.362 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:26.363 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:26.363 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:26.363 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:26.363 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:26.807 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-26T01:22:26.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-26T01:22:26.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T01:22:26.906 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T01:22:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:26.918 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:26.918 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-26T01:22:26.971 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:26.971 INFO:tasks.ceph:role: osd.4 2024-04-26T01:22:26.971 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:22:26.971 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:27.037 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:27.500 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-26T01:22:27.502 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-26T01:22:27.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-26T01:22:27.648 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-26T01:22:27.659 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:27.660 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-04-26T01:22:27.717 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:27.717 INFO:tasks.ceph:role: osd.8 2024-04-26T01:22:27.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:22:27.717 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:22:27.780 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:27.781 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:28.265 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-26T01:22:28.267 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-26T01:22:28.267 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-04-26T01:22:28.427 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-04-26T01:22:28.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:28.439 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:28.439 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-04-26T01:22:28.496 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:28.496 INFO:tasks.ceph:role: osd.12 2024-04-26T01:22:28.496 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:22:28.496 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:28.560 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:28.975 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-26T01:22:28.977 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-26T01:22:28.977 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-04-26T01:22:29.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-04-26T01:22:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:29.084 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:29.084 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:29.196 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:29.196+0000 7f9938fed7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T01:22:29.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:29.196+0000 7f9938fed7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T01:22:29.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:29.196+0000 7f9938fed7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T01:22:29.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:29.196+0000 7f9938fed7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T01:22:29.199 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:29.196+0000 7f9938fed7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T01:22:29.200 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:29.200+0000 7f9938fed7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T01:22:32.185 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T01:22:32.200 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:32.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:32.280+0000 7fd42d0c27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-26T01:22:32.283 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:32.284+0000 7fd42d0c27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-26T01:22:32.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:32.284+0000 7fd42d0c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T01:22:32.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:32.284+0000 7fd42d0c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T01:22:32.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:32.284+0000 7fd42d0c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T01:22:32.284 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:32.284+0000 7fd42d0c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-26T01:22:35.271 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-26T01:22:35.325 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:35.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:35.408+0000 7fa35c0fc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-04-26T01:22:35.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:35.408+0000 7fa35c0fc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-04-26T01:22:35.408 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:35.408+0000 7fa35c0fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-26T01:22:35.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:35.408+0000 7fa35c0fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-26T01:22:35.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:35.408+0000 7fa35c0fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2024-04-26T01:22:35.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:35.408+0000 7fa35c0fc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-04-26T01:22:38.375 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-04-26T01:22:38.432 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:38.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:38.512+0000 7fe53ff837c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-04-26T01:22:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:38.512+0000 7fe53ff837c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-04-26T01:22:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:38.512+0000 7fe53ff837c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-26T01:22:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:38.512+0000 7fe53ff837c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-26T01:22:38.515 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:38.512+0000 7fe53ff837c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-12/block: (2) No such file or directory 2024-04-26T01:22:38.516 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-26T01:22:38.516+0000 7fe53ff837c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-04-26T01:22:41.487 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-04-26T01:22:41.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T01:22:41.557 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:41.557 INFO:tasks.ceph:role: osd.1 2024-04-26T01:22:41.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:22:41.557 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:41.613 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:41.636 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-26T01:22:41.638 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-26T01:22:41.638 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-04-26T01:22:41.700 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T01:22:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:41.755 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:41.756 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-26T01:22:41.809 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:41.809 INFO:tasks.ceph:role: osd.5 2024-04-26T01:22:41.809 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:22:41.809 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:41.864 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:41.865 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:41.887 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-26T01:22:41.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-26T01:22:41.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-26T01:22:41.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-26T01:22:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:42.002 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:42.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-04-26T01:22:42.056 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:42.056 INFO:tasks.ceph:role: osd.9 2024-04-26T01:22:42.056 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:22:42.056 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:22:42.113 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:42.113 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:42.114 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:42.136 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-26T01:22:42.138 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-26T01:22:42.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-04-26T01:22:42.197 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-04-26T01:22:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:42.250 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:42.250 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-04-26T01:22:42.308 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:42.309 INFO:tasks.ceph:role: osd.13 2024-04-26T01:22:42.309 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:22:42.309 DEBUG:teuthology.orchestra.run.smithi045:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:22:42.365 INFO:teuthology.orchestra.run.smithi045.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:42.365 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:42.365 INFO:teuthology.orchestra.run.smithi045.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:42.365 INFO:teuthology.orchestra.run.smithi045.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:42.365 INFO:teuthology.orchestra.run.smithi045.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:42.365 INFO:teuthology.orchestra.run.smithi045.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:42.366 INFO:teuthology.orchestra.run.smithi045.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:42.366 INFO:teuthology.orchestra.run.smithi045.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:42.366 INFO:teuthology.orchestra.run.smithi045.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:42.366 INFO:teuthology.orchestra.run.smithi045.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:42.389 INFO:teuthology.orchestra.run.smithi045.stdout:Discarding blocks...Done. 2024-04-26T01:22:42.390 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi045.front.sepia.ceph.com -o noatime 2024-04-26T01:22:42.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-04-26T01:22:42.449 DEBUG:teuthology.orchestra.run.smithi045:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-04-26T01:22:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:42.498 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:42.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:42.611+0000 7f30a4c687c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T01:22:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:42.611+0000 7f30a4c687c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T01:22:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:42.611+0000 7f30a4c687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T01:22:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:42.611+0000 7f30a4c687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T01:22:42.605 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:42.611+0000 7f30a4c687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T01:22:42.606 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:42.611+0000 7f30a4c687c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T01:22:45.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T01:22:45.621 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:45.707+0000 7f150a9aa7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-26T01:22:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:45.707+0000 7f150a9aa7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-26T01:22:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:45.707+0000 7f150a9aa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T01:22:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:45.711+0000 7f150a9aa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T01:22:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:45.711+0000 7f150a9aa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T01:22:45.705 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:45.711+0000 7f150a9aa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-26T01:22:48.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-26T01:22:48.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:48.847+0000 7f312d2507c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-04-26T01:22:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:48.847+0000 7f312d2507c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-04-26T01:22:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:48.847+0000 7f312d2507c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-26T01:22:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:48.847+0000 7f312d2507c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-26T01:22:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:48.847+0000 7f312d2507c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2024-04-26T01:22:48.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:48.847+0000 7f312d2507c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-04-26T01:22:51.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-04-26T01:22:51.874 DEBUG:teuthology.orchestra.run.smithi045:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:51.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:51.959+0000 7ff462dfe7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-04-26T01:22:51.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:51.959+0000 7ff462dfe7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-04-26T01:22:51.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:51.959+0000 7ff462dfe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-26T01:22:51.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:51.959+0000 7ff462dfe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-26T01:22:51.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:51.959+0000 7ff462dfe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-13/block: (2) No such file or directory 2024-04-26T01:22:51.957 INFO:teuthology.orchestra.run.smithi045.stderr:2024-04-26T01:22:51.963+0000 7ff462dfe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-04-26T01:22:54.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-04-26T01:22:54.974 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T01:22:54.989 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:54.989 INFO:tasks.ceph:role: osd.2 2024-04-26T01:22:54.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:22:54.989 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:22:55.050 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:55.050 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:55.051 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:55.051 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:55.051 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:55.051 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:55.051 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:55.499 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-26T01:22:55.501 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-26T01:22:55.501 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-04-26T01:22:55.597 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T01:22:55.606 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:55.607 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-26T01:22:55.660 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:55.660 INFO:tasks.ceph:role: osd.6 2024-04-26T01:22:55.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:22:55.660 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:22:55.722 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:55.723 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:56.194 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-26T01:22:56.197 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-26T01:22:56.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-04-26T01:22:56.333 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-26T01:22:56.344 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:56.345 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-04-26T01:22:56.403 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:56.403 INFO:tasks.ceph:role: osd.10 2024-04-26T01:22:56.403 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:22:56.403 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:22:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:56.467 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:56.467 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:56.467 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:56.467 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:56.467 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:56.468 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:56.468 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:56.468 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:56.951 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-26T01:22:56.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-26T01:22:56.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-04-26T01:22:57.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-04-26T01:22:57.123 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:57.124 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-04-26T01:22:57.178 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-04-26T01:22:57.182 INFO:tasks.ceph:role: osd.14 2024-04-26T01:22:57.182 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:22:57.182 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:22:57.242 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:22:57.243 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:22:57.670 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-04-26T01:22:57.671 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-04-26T01:22:57.672 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-04-26T01:22:57.774 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-04-26T01:22:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:22:57.785 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:22:57.785 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:22:57.920 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:22:57.922+0000 7fe7fa15f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T01:22:57.926 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:22:57.926+0000 7fe7fa15f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T01:22:57.926 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:22:57.926+0000 7fe7fa15f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T01:22:57.926 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:22:57.926+0000 7fe7fa15f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T01:22:57.926 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:22:57.926+0000 7fe7fa15f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T01:22:57.926 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:22:57.930+0000 7fe7fa15f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T01:23:00.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T01:23:00.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:01.032 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:01.034+0000 7fd5c9ce47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-26T01:23:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:01.034+0000 7fd5c9ce47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-26T01:23:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:01.034+0000 7fd5c9ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T01:23:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:01.034+0000 7fd5c9ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T01:23:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:01.034+0000 7fd5c9ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-26T01:23:01.033 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:01.034+0000 7fd5c9ce47c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-26T01:23:04.015 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T01:23:04.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:04.157 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:04.158+0000 7f89c98ca7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-04-26T01:23:04.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:04.158+0000 7f89c98ca7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-04-26T01:23:04.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:04.158+0000 7f89c98ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-26T01:23:04.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:04.158+0000 7f89c98ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-26T01:23:04.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:04.158+0000 7f89c98ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-10/block: (2) No such file or directory 2024-04-26T01:23:04.158 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:04.158+0000 7f89c98ca7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-04-26T01:23:07.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-04-26T01:23:07.197 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:07.276 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:07.278+0000 7fc9c546b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-04-26T01:23:07.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:07.278+0000 7fc9c546b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-04-26T01:23:07.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:07.278+0000 7fc9c546b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-26T01:23:07.277 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:07.278+0000 7fc9c546b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-26T01:23:07.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:07.278+0000 7fc9c546b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-14/block: (2) No such file or directory 2024-04-26T01:23:07.278 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-26T01:23:07.278+0000 7fc9c546b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-04-26T01:23:10.262 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-04-26T01:23:10.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T01:23:10.334 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:23:10.334 INFO:tasks.ceph:role: osd.3 2024-04-26T01:23:10.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:23:10.334 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T01:23:10.397 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:23:10.397 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:23:10.398 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:23:10.861 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-04-26T01:23:10.863 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-04-26T01:23:10.863 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-26T01:23:11.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T01:23:11.041 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:23:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:11.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-26T01:23:11.101 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:23:11.101 INFO:tasks.ceph:role: osd.7 2024-04-26T01:23:11.102 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:23:11.102 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T01:23:11.166 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:23:11.166 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:23:11.166 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:23:11.166 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:23:11.167 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:23:11.167 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:23:11.167 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:23:11.167 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:23:11.167 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:23:11.167 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:23:11.603 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-04-26T01:23:11.605 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-04-26T01:23:11.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-04-26T01:23:11.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-26T01:23:11.743 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:23:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:11.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-04-26T01:23:11.798 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:23:11.798 INFO:tasks.ceph:role: osd.11 2024-04-26T01:23:11.798 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:23:11.798 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T01:23:11.861 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:23:11.862 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:23:12.320 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-04-26T01:23:12.322 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-04-26T01:23:12.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-04-26T01:23:12.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-04-26T01:23:12.478 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:23:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:12.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-04-26T01:23:12.534 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-04-26T01:23:12.534 INFO:tasks.ceph:role: osd.15 2024-04-26T01:23:12.534 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:23:12.534 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T01:23:12.598 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T01:23:12.598 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T01:23:12.598 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T01:23:12.598 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T01:23:12.599 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T01:23:12.599 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-04-26T01:23:12.599 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T01:23:12.599 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T01:23:12.599 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T01:23:12.599 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T01:23:13.090 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-04-26T01:23:13.093 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-04-26T01:23:13.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-04-26T01:23:13.231 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-04-26T01:23:13.242 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T01:23:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:23:13.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:13.359 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:13.364+0000 7f893a3877c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T01:23:13.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:13.368+0000 7f893a3877c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T01:23:13.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:13.368+0000 7f893a3877c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T01:23:13.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:13.368+0000 7f893a3877c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T01:23:13.362 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:13.368+0000 7f893a3877c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T01:23:13.363 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:13.368+0000 7f893a3877c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T01:23:16.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T01:23:16.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:16.420 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:16.424+0000 7fef94b347c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-26T01:23:16.421 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:16.424+0000 7fef94b347c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-26T01:23:16.421 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:16.424+0000 7fef94b347c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T01:23:16.421 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:16.424+0000 7fef94b347c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T01:23:16.421 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:16.424+0000 7fef94b347c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-26T01:23:16.421 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:16.424+0000 7fef94b347c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-26T01:23:19.394 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-26T01:23:19.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:19.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:19.536+0000 7f1d37c3e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-04-26T01:23:19.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:19.536+0000 7f1d37c3e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-04-26T01:23:19.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:19.536+0000 7f1d37c3e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-26T01:23:19.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:19.536+0000 7f1d37c3e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-26T01:23:19.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:19.536+0000 7f1d37c3e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-11/block: (2) No such file or directory 2024-04-26T01:23:19.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:19.536+0000 7f1d37c3e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-04-26T01:23:22.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-04-26T01:23:22.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:22.672 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:22.676+0000 7fe93d3237c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-04-26T01:23:22.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:22.676+0000 7fe93d3237c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-04-26T01:23:22.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:22.676+0000 7fe93d3237c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-26T01:23:22.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:22.676+0000 7fe93d3237c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-26T01:23:22.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:22.676+0000 7fe93d3237c0 -1 bluestore(/var/lib/ceph/osd/ceph-15/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-15/block: (2) No such file or directory 2024-04-26T01:23:22.674 INFO:teuthology.orchestra.run.smithi181.stderr:2024-04-26T01:23:22.680+0000 7fe93d3237c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-04-26T01:23:25.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-04-26T01:23:25.735 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T01:23:25.736 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:25.736 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-26T01:23:25.751 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:25.751 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T01:23:25.808 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:25.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-26T01:23:25.863 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:25.864 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-04-26T01:23:25.919 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:25.919 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-04-26T01:23:25.975 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:25.975 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T01:23:25.990 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:25.990 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-26T01:23:26.045 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:26.046 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-04-26T01:23:26.100 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:26.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-04-26T01:23:26.156 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:26.156 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T01:23:26.171 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:26.171 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-26T01:23:26.226 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:26.227 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-04-26T01:23:26.282 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:26.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-04-26T01:23:26.334 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:26.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T01:23:26.348 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:26.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T01:23:26.402 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:26.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-26T01:23:26.458 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:26.458 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-04-26T01:23:26.509 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:26.510 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-04-26T01:23:26.561 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:26.561 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-26T01:23:26.607 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T01:23:26.608 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T01:23:26.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T01:23:26.614 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T01:23:26.622 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.y] 2024-04-26T01:23:26.625 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDSAStmKuMSCBAArEUWXR3wQHYconpurqfPBg== 2024-04-26T01:23:26.625 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-04-26T01:23:26.625 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDVAStm/9H9CxAAjqkUdkHdQuyxPrv3Z3UeQA== 2024-04-26T01:23:26.625 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-04-26T01:23:26.625 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDYAStmgVcpERAARRSmB0YiT7YPshTRC9nPNQ== 2024-04-26T01:23:26.625 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.8] 2024-04-26T01:23:26.625 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDbAStm5p+ZGBAAiZKD5k/DgflYy4L6k0r5oQ== 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.12] 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDeAStmlQj4HhAAi6f9D6YVkcY37B45p0M5AA== 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDiAStmq1F/JBAAvgvxUTY08u9DLaVFLjMEzw== 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDlAStmovBmKhAAXUxHjuwmxKiTgTSDPb79oQ== 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.9] 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDoAStmgEiiMhAArOXcm1eBvzmuia9nVFNgAg== 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.13] 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDrAStmqYlrORAAn1H9V6AXr3hVtQ2Tpo4zeg== 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-04-26T01:23:26.626 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDxAStmjFNUNxAAcEJ1RmSAKphQ91a54J9lYQ== 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.6] 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQD1AStmnKhfAhAAshtGpF/iIe+MCOMoDpOHrA== 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.10] 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQD4AStmY2jQCRAARNqhXNbJ5JgE8eocwSJxvA== 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.14] 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQD7AStmSNvyEBAAESF3p9/tqiy25xIFsfP/MQ== 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDSAStmbpfxChAA/KA5Y+CCSnD8pnAfeiePRA== 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQABAitmjkjkFRAAjs8cv/QyrYf0hIYjnRziIw== 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.7] 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAEAitmboiHGRAAxs0Owkt7znDFDPZkW/1ywA== 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.11] 2024-04-26T01:23:26.627 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAHAitmqDsuIBAAr2iG7RW+yn9K/3EQ02EXTA== 2024-04-26T01:23:26.628 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.15] 2024-04-26T01:23:26.628 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAKAitmkf2MKBAANFpj9d6TOq8Q3Rd3F9soDg== 2024-04-26T01:23:26.628 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-04-26T01:23:26.628 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQDSAStmXEyhEBAAT5eHYUdQPJsFJoCcat/CeA== 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDSAStmKuMSCBAArEUWXR3wQHYconpurqfPBg== 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDVAStm/9H9CxAAjqkUdkHdQuyxPrv3Z3UeQA== 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDYAStmgVcpERAARRSmB0YiT7YPshTRC9nPNQ== 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.8] 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDbAStm5p+ZGBAAiZKD5k/DgflYy4L6k0r5oQ== 2024-04-26T01:23:26.630 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.12] 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDeAStmlQj4HhAAi6f9D6YVkcY37B45p0M5AA== 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDiAStmq1F/JBAAvgvxUTY08u9DLaVFLjMEzw== 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDlAStmovBmKhAAXUxHjuwmxKiTgTSDPb79oQ== 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.9] 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDoAStmgEiiMhAArOXcm1eBvzmuia9nVFNgAg== 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.13] 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDrAStmqYlrORAAn1H9V6AXr3hVtQ2Tpo4zeg== 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDxAStmjFNUNxAAcEJ1RmSAKphQ91a54J9lYQ== 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQD1AStmnKhfAhAAshtGpF/iIe+MCOMoDpOHrA== 2024-04-26T01:23:26.631 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.10] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQD4AStmY2jQCRAARNqhXNbJ5JgE8eocwSJxvA== 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.14] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQD7AStmSNvyEBAAESF3p9/tqiy25xIFsfP/MQ== 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDSAStmbpfxChAA/KA5Y+CCSnD8pnAfeiePRA== 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQABAitmjkjkFRAAjs8cv/QyrYf0hIYjnRziIw== 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAEAitmboiHGRAAxs0Owkt7znDFDPZkW/1ywA== 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.11] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAHAitmqDsuIBAAr2iG7RW+yn9K/3EQ02EXTA== 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.15] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQAKAitmkf2MKBAANFpj9d6TOq8Q3Rd3F9soDg== 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-04-26T01:23:26.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDSAStmXEyhEBAAT5eHYUdQPJsFJoCcat/CeA== 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.y] 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDSAStmKuMSCBAArEUWXR3wQHYconpurqfPBg== 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.0] 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDVAStm/9H9CxAAjqkUdkHdQuyxPrv3Z3UeQA== 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.4] 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDYAStmgVcpERAARRSmB0YiT7YPshTRC9nPNQ== 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.8] 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDbAStm5p+ZGBAAiZKD5k/DgflYy4L6k0r5oQ== 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.12] 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDeAStmlQj4HhAAi6f9D6YVkcY37B45p0M5AA== 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.1] 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDiAStmq1F/JBAAvgvxUTY08u9DLaVFLjMEzw== 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.5] 2024-04-26T01:23:26.633 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDlAStmovBmKhAAXUxHjuwmxKiTgTSDPb79oQ== 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.9] 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDoAStmgEiiMhAArOXcm1eBvzmuia9nVFNgAg== 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.13] 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDrAStmqYlrORAAn1H9V6AXr3hVtQ2Tpo4zeg== 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.2] 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDxAStmjFNUNxAAcEJ1RmSAKphQ91a54J9lYQ== 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.6] 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQD1AStmnKhfAhAAshtGpF/iIe+MCOMoDpOHrA== 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.10] 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQD4AStmY2jQCRAARNqhXNbJ5JgE8eocwSJxvA== 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.14] 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQD7AStmSNvyEBAAESF3p9/tqiy25xIFsfP/MQ== 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout:[mgr.x] 2024-04-26T01:23:26.634 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDSAStmbpfxChAA/KA5Y+CCSnD8pnAfeiePRA== 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.3] 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQABAitmjkjkFRAAjs8cv/QyrYf0hIYjnRziIw== 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.7] 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAEAitmboiHGRAAxs0Owkt7znDFDPZkW/1ywA== 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.11] 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAHAitmqDsuIBAAr2iG7RW+yn9K/3EQ02EXTA== 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout:[osd.15] 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQAKAitmkf2MKBAANFpj9d6TOq8Q3Rd3F9soDg== 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-04-26T01:23:26.635 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQDSAStmXEyhEBAAT5eHYUdQPJsFJoCcat/CeA== 2024-04-26T01:23:26.635 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:23:26.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:23:26.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:23:26.720 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.808 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.895 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.934 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:26.988 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.018 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.061 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.068 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.102 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.145 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.229 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.232 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.274 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.315 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.318 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.358 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.403 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.408 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.492 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.498 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.574 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.580 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.657 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.669 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.741 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:23:27.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:23:27.758 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T01:23:27.823 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.841 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.843 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.994 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:27.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:28.077 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:28.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:28.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T01:23:28.161 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T01:23:28.164 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T01:23:28.166 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T01:23:28.247 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T01:23:28.247 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T01:23:28.264 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T01:23:28.357 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T01:23:28.369 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T01:23:28.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T01:23:28.480 DEBUG:teuthology.orchestra.run.smithi045:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T01:23:28.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T01:23:28.506 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T01:23:28.602 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T01:23:28.614 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:28.616 DEBUG:teuthology.orchestra.run.smithi045:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:28.619 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T01:23:28.659 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T01:23:28.659 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T01:23:28.659 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T01:23:28.664 INFO:tasks.ceph.mon.a:Started 2024-04-26T01:23:28.665 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T01:23:28.665 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T01:23:28.667 INFO:tasks.ceph.mon.b:Started 2024-04-26T01:23:28.667 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T01:23:28.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T01:23:28.704 INFO:tasks.ceph.mon.c:Started 2024-04-26T01:23:28.704 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T01:23:28.705 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-26T01:23:28.705 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-26T01:23:28.707 INFO:tasks.ceph.mgr.y:Started 2024-04-26T01:23:28.707 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T01:23:28.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T01:23:28.710 INFO:tasks.ceph.mgr.x:Started 2024-04-26T01:23:28.710 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T01:23:28.710 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T01:23:35.204 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-04-26T01:23:35.224 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T01:23:35.224 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T01:23:35.224 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:35.224 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T01:23:35.236 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:35.236 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-26T01:23:35.291 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:35.291 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-04-26T01:23:35.303 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:35.308+0000 7f1096e3f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:23:35.314 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-26T01:23:35.314 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-04-26T01:23:35.366 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:35.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T01:23:35.381 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:35.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-26T01:23:35.437 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:35.437 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-04-26T01:23:35.442 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:35.443+0000 7f0c9d929180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T01:23:35.492 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-04-26T01:23:35.492 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-04-26T01:23:35.549 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:35.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T01:23:35.563 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:35.564 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-26T01:23:35.619 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:35.619 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-04-26T01:23:35.675 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-26T01:23:35.675 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-04-26T01:23:35.731 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:35.731 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T01:23:35.744 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:35.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-26T01:23:35.798 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:35.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-04-26T01:23:35.818 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:35.820+0000 7f1096e3f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:23:35.854 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-04-26T01:23:35.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-04-26T01:23:35.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new a5592650-ee45-45a1-94c4-487000bdb588 0 2024-04-26T01:23:35.930 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:35.931+0000 7f0c9d929180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T01:23:36.012 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.016+0000 7f1096e3f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:23:36.120 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:36.123+0000 7f0c9d929180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T01:23:36.133 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-04-26T01:23:36.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 26af5691-fc53-4b20-9c35-012d299e978b 1 2024-04-26T01:23:36.338 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-04-26T01:23:36.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new ba3d860b-252b-42de-a951-99329413b1bf 2 2024-04-26T01:23:36.550 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-04-26T01:23:36.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 8294850e-c851-4be8-adb0-ffe9ef4fd827 3 2024-04-26T01:23:36.581 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.584+0000 7f1096e3f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:23:36.636 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.640+0000 7f1096e3f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:23:36.682 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:36.683+0000 7f0c9d929180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T01:23:36.691 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.696+0000 7f1096e3f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:23:36.737 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:36.739+0000 7f0c9d929180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T01:23:36.764 INFO:teuthology.orchestra.run.smithi181.stdout:3 2024-04-26T01:23:36.773 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.776+0000 7f1096e3f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:23:36.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 601a444c-5cc7-4ae1-9cd3-379b6d7155da 4 2024-04-26T01:23:36.792 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:36.795+0000 7f0c9d929180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T01:23:36.827 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.832+0000 7f1096e3f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:23:36.871 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:36.875+0000 7f0c9d929180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T01:23:36.907 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.912+0000 7f1096e3f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:23:36.925 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:36.927+0000 7f0c9d929180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T01:23:36.963 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:36.964+0000 7f1096e3f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:23:36.989 INFO:teuthology.orchestra.run.smithi181.stdout:4 2024-04-26T01:23:37.004 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:37.007+0000 7f0c9d929180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T01:23:37.005 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 6818ea43-206d-49df-9df8-53d9ff132f8e 5 2024-04-26T01:23:37.058 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:37.059+0000 7f0c9d929180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T01:23:37.121 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:37.124+0000 7f1096e3f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:23:37.186 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:37.188+0000 7f1096e3f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:23:37.215 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:37.219+0000 7f0c9d929180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T01:23:37.245 INFO:teuthology.orchestra.run.smithi181.stdout:5 2024-04-26T01:23:37.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 7e47d5e9-00aa-428f-8b6c-256af34a821c 6 2024-04-26T01:23:37.278 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:37.279+0000 7f0c9d929180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T01:23:37.484 INFO:teuthology.orchestra.run.smithi181.stdout:6 2024-04-26T01:23:37.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 0ed4ac32-3700-4879-abde-e664c307dcdf 7 2024-04-26T01:23:37.724 INFO:teuthology.orchestra.run.smithi181.stdout:7 2024-04-26T01:23:37.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 94604649-8873-4348-a27c-47e6751274c4 8 2024-04-26T01:23:37.829 INFO:tasks.ceph.mgr.y.smithi037.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T01:23:37.830 INFO:tasks.ceph.mgr.y.smithi037.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T01:23:37.830 INFO:tasks.ceph.mgr.y.smithi037.stderr: from numpy import show_config as show_numpy_config 2024-04-26T01:23:37.887 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:37.892+0000 7f1096e3f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:23:37.945 INFO:teuthology.orchestra.run.smithi181.stdout:8 2024-04-26T01:23:37.962 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new cf4159cb-22fe-49c9-b94c-36a6003e6c3b 9 2024-04-26T01:23:37.970 INFO:tasks.ceph.mgr.x.smithi181.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T01:23:37.970 INFO:tasks.ceph.mgr.x.smithi181.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T01:23:37.970 INFO:tasks.ceph.mgr.x.smithi181.stderr: from numpy import show_config as show_numpy_config 2024-04-26T01:23:38.038 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:38.040+0000 7f1096e3f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:23:38.042 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:38.043+0000 7f0c9d929180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T01:23:38.158 INFO:teuthology.orchestra.run.smithi181.stdout:9 2024-04-26T01:23:38.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new bf42aad2-b60d-45a4-adde-f258ce7086b3 10 2024-04-26T01:23:38.204 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:38.208+0000 7f1096e3f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:23:38.210 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:38.211+0000 7f0c9d929180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:23:38.270 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:38.272+0000 7f1096e3f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:23:38.382 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:38.383+0000 7f0c9d929180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T01:23:38.411 INFO:teuthology.orchestra.run.smithi181.stdout:10 2024-04-26T01:23:38.412 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:38.416+0000 7f1096e3f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:23:38.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 1db3483a-2bc5-4cb5-9034-224068e68cdd 11 2024-04-26T01:23:38.453 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:38.455+0000 7f0c9d929180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T01:23:38.597 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:38.599+0000 7f0c9d929180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T01:23:38.621 INFO:teuthology.orchestra.run.smithi181.stdout:11 2024-04-26T01:23:38.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 970dc59e-69c3-4a15-ada5-569b3ee520b6 12 2024-04-26T01:23:38.873 INFO:teuthology.orchestra.run.smithi181.stdout:12 2024-04-26T01:23:38.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new cae6ef09-e4e5-4d6c-8934-065720075a50 13 2024-04-26T01:23:39.053 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:39.056+0000 7f1096e3f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:23:39.087 INFO:teuthology.orchestra.run.smithi181.stdout:13 2024-04-26T01:23:39.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 02493b55-1966-4731-8bba-891cfb8f2244 14 2024-04-26T01:23:39.132 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:39.136+0000 7f1096e3f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:23:39.245 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:39.247+0000 7f0c9d929180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T01:23:39.318 INFO:teuthology.orchestra.run.smithi181.stdout:14 2024-04-26T01:23:39.321 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:39.323+0000 7f0c9d929180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T01:23:39.322 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:39.324+0000 7f1096e3f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:23:39.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 93d2b85d-c322-44b6-aa6a-c5da975ba45a 15 2024-04-26T01:23:39.399 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:39.404+0000 7f1096e3f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:23:39.509 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:39.511+0000 7f0c9d929180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T01:23:39.529 INFO:teuthology.orchestra.run.smithi181.stdout:15 2024-04-26T01:23:39.545 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T01:23:39.545 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T01:23:39.547 INFO:tasks.ceph.osd.0:Started 2024-04-26T01:23:39.548 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-26T01:23:39.548 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-26T01:23:39.550 INFO:tasks.ceph.osd.4:Started 2024-04-26T01:23:39.550 INFO:tasks.ceph.osd.8:Restarting daemon 2024-04-26T01:23:39.551 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-04-26T01:23:39.553 INFO:tasks.ceph.osd.8:Started 2024-04-26T01:23:39.553 INFO:tasks.ceph.osd.12:Restarting daemon 2024-04-26T01:23:39.553 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-04-26T01:23:39.555 INFO:tasks.ceph.osd.12:Started 2024-04-26T01:23:39.556 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T01:23:39.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T01:23:39.558 INFO:tasks.ceph.osd.1:Started 2024-04-26T01:23:39.559 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-26T01:23:39.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-26T01:23:39.561 INFO:tasks.ceph.osd.5:Started 2024-04-26T01:23:39.561 INFO:tasks.ceph.osd.9:Restarting daemon 2024-04-26T01:23:39.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-04-26T01:23:39.564 INFO:tasks.ceph.osd.9:Started 2024-04-26T01:23:39.564 INFO:tasks.ceph.osd.13:Restarting daemon 2024-04-26T01:23:39.564 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-04-26T01:23:39.566 INFO:tasks.ceph.osd.13:Started 2024-04-26T01:23:39.566 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T01:23:39.567 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T01:23:39.569 INFO:tasks.ceph.osd.2:Started 2024-04-26T01:23:39.569 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-26T01:23:39.569 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-26T01:23:39.572 INFO:tasks.ceph.osd.6:Started 2024-04-26T01:23:39.572 INFO:tasks.ceph.osd.10:Restarting daemon 2024-04-26T01:23:39.572 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-04-26T01:23:39.574 INFO:tasks.ceph.osd.10:Started 2024-04-26T01:23:39.574 INFO:tasks.ceph.osd.14:Restarting daemon 2024-04-26T01:23:39.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-04-26T01:23:39.577 INFO:tasks.ceph.osd.14:Started 2024-04-26T01:23:39.577 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T01:23:39.577 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T01:23:39.579 INFO:tasks.ceph.osd.3:Started 2024-04-26T01:23:39.579 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-26T01:23:39.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-26T01:23:39.581 INFO:tasks.ceph.osd.7:Started 2024-04-26T01:23:39.581 INFO:tasks.ceph.osd.11:Restarting daemon 2024-04-26T01:23:39.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-04-26T01:23:39.582 INFO:tasks.ceph.osd.11:Started 2024-04-26T01:23:39.583 INFO:tasks.ceph.osd.15:Restarting daemon 2024-04-26T01:23:39.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-04-26T01:23:39.584 INFO:tasks.ceph.osd.15:Started 2024-04-26T01:23:39.585 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:23:39.586 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:39.587+0000 7f0c9d929180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T01:23:39.719 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:39.720+0000 7f1096e3f180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:23:39.775 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:39.776 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":18,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:23:39.528016+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T01:23:39.528016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:23:39.784 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:39.788+0000 7f1096e3f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:23:39.791 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T01:23:39.791 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T01:23:39.901 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:39.903+0000 7f0c9d929180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T01:23:39.962 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-04-26T01:23:39.963+0000 7f0c9d929180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T01:23:40.177 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:23:40.180+0000 7f7d0bbab7c0 -1 Falling back to public interface 2024-04-26T01:23:40.177 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:23:40.181+0000 7f4a53cd37c0 -1 Falling back to public interface 2024-04-26T01:23:40.180 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:23:40.185+0000 7fb27cefd7c0 -1 Falling back to public interface 2024-04-26T01:23:40.181 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:23:40.184+0000 7f9831c2c7c0 -1 Falling back to public interface 2024-04-26T01:23:40.182 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:23:40.185+0000 7f22fa0627c0 -1 Falling back to public interface 2024-04-26T01:23:40.186 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:23:40.189+0000 7faa62b147c0 -1 Falling back to public interface 2024-04-26T01:23:40.187 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:23:40.189+0000 7f04f8d5a7c0 -1 Falling back to public interface 2024-04-26T01:23:40.187 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:23:40.189+0000 7f76a7a287c0 -1 Falling back to public interface 2024-04-26T01:23:40.188 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:23:40.193+0000 7efdd82187c0 -1 Falling back to public interface 2024-04-26T01:23:40.196 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:23:40.200+0000 7f23b1e6f7c0 -1 Falling back to public interface 2024-04-26T01:23:40.196 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:23:40.200+0000 7ff35dbfc7c0 -1 Falling back to public interface 2024-04-26T01:23:40.197 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:23:40.201+0000 7f3b173527c0 -1 Falling back to public interface 2024-04-26T01:23:40.213 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:23:40.215+0000 7f487f6d37c0 -1 Falling back to public interface 2024-04-26T01:23:40.214 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:23:40.215+0000 7f8645c7e7c0 -1 Falling back to public interface 2024-04-26T01:23:40.215 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:40.215+0000 7fef2cfeb7c0 -1 Falling back to public interface 2024-04-26T01:23:40.215 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:23:40.215+0000 7f8644c3e640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6806/0: (98) Address already in use 2024-04-26T01:23:40.215 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:23:40.215+0000 7f487e693640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6809/0: (98) Address already in use 2024-04-26T01:23:40.223 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:23:40.227+0000 7f1ff7a947c0 -1 Falling back to public interface 2024-04-26T01:23:40.824 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:40.828+0000 7f103e189640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T01:23:40.827 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-04-26T01:23:40.828+0000 7f103e189640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T01:23:41.403 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T01:23:41.707 INFO:teuthology.misc.health.smithi037.stdout: 2024-04-26T01:23:41.707 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":18,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:23:39.528016+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T01:23:39.528016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:23:41.724 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-04-26T01:23:43.830 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:43.830+0000 7fef2cfeb7c0 -1 osd.15 0 log_to_monitors true 2024-04-26T01:23:43.858 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:23:43.860+0000 7f7d0bbab7c0 -1 osd.4 0 log_to_monitors true 2024-04-26T01:23:43.953 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:23:43.956+0000 7f9831c2c7c0 -1 osd.8 0 log_to_monitors true 2024-04-26T01:23:43.953 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:23:43.954+0000 7f8645c7e7c0 -1 osd.7 0 log_to_monitors true 2024-04-26T01:23:43.971 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:23:43.973+0000 7efdd82187c0 -1 osd.10 0 log_to_monitors true 2024-04-26T01:23:44.004 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:23:44.005+0000 7f04f8d5a7c0 -1 osd.13 0 log_to_monitors true 2024-04-26T01:23:44.015 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:23:44.017+0000 7f22fa0627c0 -1 osd.5 0 log_to_monitors true 2024-04-26T01:23:44.022 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:23:44.024+0000 7ff35dbfc7c0 -1 osd.0 0 log_to_monitors true 2024-04-26T01:23:44.060 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:23:44.065+0000 7fb27cefd7c0 -1 osd.6 0 log_to_monitors true 2024-04-26T01:23:44.061 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:23:44.064+0000 7f23b1e6f7c0 -1 osd.12 0 log_to_monitors true 2024-04-26T01:23:44.062 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:23:44.065+0000 7f4a53cd37c0 -1 osd.2 0 log_to_monitors true 2024-04-26T01:23:44.113 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:23:44.114+0000 7f1ff7a947c0 -1 osd.11 0 log_to_monitors true 2024-04-26T01:23:44.113 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:23:44.114+0000 7f487f6d37c0 -1 osd.3 0 log_to_monitors true 2024-04-26T01:23:44.127 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:23:44.129+0000 7f76a7a287c0 -1 osd.1 0 log_to_monitors true 2024-04-26T01:23:44.134 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:23:44.137+0000 7faa62b147c0 -1 osd.9 0 log_to_monitors true 2024-04-26T01:23:44.163 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:23:44.165+0000 7f3b173527c0 -1 osd.14 0 log_to_monitors true 2024-04-26T01:23:45.851 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:23:45.853+0000 7f76a39d1640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T01:23:45.851 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:23:45.853+0000 7faa5eabd640 -1 osd.9 0 waiting for initial osdmap 2024-04-26T01:23:45.852 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:23:45.854+0000 7f8641c27640 -1 osd.7 0 waiting for initial osdmap 2024-04-26T01:23:45.853 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:23:45.856+0000 7f7d07b54640 -1 osd.4 0 waiting for initial osdmap 2024-04-26T01:23:45.854 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:23:45.854+0000 7f1ff3a3d640 -1 osd.11 0 waiting for initial osdmap 2024-04-26T01:23:45.855 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:23:45.857+0000 7fb278ea6640 -1 osd.6 0 waiting for initial osdmap 2024-04-26T01:23:45.855 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:23:45.857+0000 7efdd41c1640 -1 osd.10 0 waiting for initial osdmap 2024-04-26T01:23:45.855 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:23:45.857+0000 7f4a4fc7c640 -1 osd.2 0 waiting for initial osdmap 2024-04-26T01:23:45.855 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:23:45.857+0000 7f3b132fb640 -1 osd.14 0 waiting for initial osdmap 2024-04-26T01:23:45.863 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:23:45.865+0000 7f769f80e640 -1 osd.1 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.864 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:23:45.865+0000 7faa5a8fa640 -1 osd.9 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.869 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:23:45.870+0000 7f863da64640 -1 osd.7 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.870 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:23:45.870+0000 7f1fef87a640 -1 osd.11 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.873 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:23:45.877+0000 7efdcfffe640 -1 osd.10 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.874 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:23:45.877+0000 7f3b0f138640 -1 osd.14 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.884 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:23:45.888+0000 7f7d03991640 -1 osd.4 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.885 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:23:45.889+0000 7fb274ce3640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:45.885 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:23:45.889+0000 7f4a4bab9640 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:46.608 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:46.610+0000 7fef28f94640 -1 osd.15 0 waiting for initial osdmap 2024-04-26T01:23:46.636 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:46.638+0000 7fef20dc9640 -1 osd.15 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:46.653 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:23:46.653+0000 7f04f4d03640 -1 osd.13 0 waiting for initial osdmap 2024-04-26T01:23:46.678 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:23:46.681+0000 7f04ecb38640 -1 osd.13 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:46.694 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:23:46.694+0000 7f487b67c640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T01:23:46.707 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:23:46.709+0000 7f22f681d640 -1 osd.5 0 waiting for initial osdmap 2024-04-26T01:23:46.716 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:23:46.718+0000 7f48734b1640 -1 osd.3 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:46.726 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:23:46.729+0000 7f22ede40640 -1 osd.5 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:46.834 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:23:46.836+0000 7f23ade18640 -1 osd.12 0 waiting for initial osdmap 2024-04-26T01:23:46.868 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:23:46.872+0000 7f23a5c4d640 -1 osd.12 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:46.879 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:23:46.884+0000 7f982e3e7640 -1 osd.8 0 waiting for initial osdmap 2024-04-26T01:23:46.892 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:23:46.896+0000 7ff359ba5640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T01:23:46.912 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:23:46.916+0000 7f9825a0a640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:46.944 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:23:46.948+0000 7ff3519da640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T01:23:49.338 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T01:23:49.647 INFO:teuthology.misc.health.smithi037.stdout: 2024-04-26T01:23:49.648 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":23,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:23:48.870422+0000","last_up_change":"2024-04-26T01:23:47.867650+0000","last_in_change":"2024-04-26T01:23:39.528016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:23:47.920762+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6819","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6820","nonce":906916438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6824","nonce":906916438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6829","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6831","nonce":906916438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6827","nonce":906916438}]},"public_addr":"172.21.15.37:6820/906916438","cluster_addr":"172.21.15.37:6824/906916438","heartbeat_back_addr":"172.21.15.37:6831/906916438","heartbeat_front_addr":"172.21.15.37:6827/906916438","state":["exists","up"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6813","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6816","nonce":1630954663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6823","nonce":1630954663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6831","nonce":1630954663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6828","nonce":1630954663}]},"public_addr":"172.21.15.45:6816/1630954663","cluster_addr":"172.21.15.45:6823/1630954663","heartbeat_back_addr":"172.21.15.45:6831/1630954663","heartbeat_front_addr":"172.21.15.45:6828/1630954663","state":["exists","up"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6801","nonce":2961312835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6803","nonce":2961312835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6807","nonce":2961312835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6805","nonce":2961312835}]},"public_addr":"172.21.15.71:6801/2961312835","cluster_addr":"172.21.15.71:6803/2961312835","heartbeat_back_addr":"172.21.15.71:6807/2961312835","heartbeat_front_addr":"172.21.15.71:6805/2961312835","state":["exists","up"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6801","nonce":1026435117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6806","nonce":1026435117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6821","nonce":1026435117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6814","nonce":1026435117}]},"public_addr":"172.21.15.181:6801/1026435117","cluster_addr":"172.21.15.181:6806/1026435117","heartbeat_back_addr":"172.21.15.181:6821/1026435117","heartbeat_front_addr":"172.21.15.181:6814/1026435117","state":["exists","up"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6803","nonce":1575437742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6805","nonce":1575437742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6809","nonce":1575437742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6807","nonce":1575437742}]},"public_addr":"172.21.15.37:6803/1575437742","cluster_addr":"172.21.15.37:6805/1575437742","heartbeat_back_addr":"172.21.15.37:6809/1575437742","heartbeat_front_addr":"172.21.15.37:6807/1575437742","state":["exists","up"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6801","nonce":3759230996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6803","nonce":3759230996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6807","nonce":3759230996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6805","nonce":3759230996}]},"public_addr":"172.21.15.45:6801/3759230996","cluster_addr":"172.21.15.45:6803/3759230996","heartbeat_back_addr":"172.21.15.45:6807/3759230996","heartbeat_front_addr":"172.21.15.45:6805/3759230996","state":["exists","up"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6809","nonce":76180415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6811","nonce":76180415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6815","nonce":76180415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6813","nonce":76180415}]},"public_addr":"172.21.15.71:6809/76180415","cluster_addr":"172.21.15.71:6811/76180415","heartbeat_back_addr":"172.21.15.71:6815/76180415","heartbeat_front_addr":"172.21.15.71:6813/76180415","state":["exists","up"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6807","nonce":4043929361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6812","nonce":4043929361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6823","nonce":4043929361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6820","nonce":4043929361}]},"public_addr":"172.21.15.181:6807/4043929361","cluster_addr":"172.21.15.181:6812/4043929361","heartbeat_back_addr":"172.21.15.181:6823/4043929361","heartbeat_front_addr":"172.21.15.181:6820/4043929361","state":["exists","up"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6811","nonce":4094482142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6813","nonce":4094482142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6817","nonce":4094482142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6815","nonce":4094482142}]},"public_addr":"172.21.15.37:6811/4094482142","cluster_addr":"172.21.15.37:6813/4094482142","heartbeat_back_addr":"172.21.15.37:6817/4094482142","heartbeat_front_addr":"172.21.15.37:6815/4094482142","state":["exists","up"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6810","nonce":3448396888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6815","nonce":3448396888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6824","nonce":3448396888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6817","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6819","nonce":3448396888}]},"public_addr":"172.21.15.45:6810/3448396888","cluster_addr":"172.21.15.45:6815/3448396888","heartbeat_back_addr":"172.21.15.45:6824/3448396888","heartbeat_front_addr":"172.21.15.45:6819/3448396888","state":["exists","up"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6817","nonce":896289472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6819","nonce":896289472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6823","nonce":896289472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6821","nonce":896289472}]},"public_addr":"172.21.15.71:6817/896289472","cluster_addr":"172.21.15.71:6819/896289472","heartbeat_back_addr":"172.21.15.71:6823/896289472","heartbeat_front_addr":"172.21.15.71:6821/896289472","state":["exists","up"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6825","nonce":647568313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6827","nonce":647568313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6831","nonce":647568313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6829","nonce":647568313}]},"public_addr":"172.21.15.181:6825/647568313","cluster_addr":"172.21.15.181:6827/647568313","heartbeat_back_addr":"172.21.15.181:6831/647568313","heartbeat_front_addr":"172.21.15.181:6829/647568313","state":["exists","up"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6821","nonce":3336489453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6825","nonce":3336489453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6833","nonce":3336489453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6830","nonce":3336489453}]},"public_addr":"172.21.15.37:6821/3336489453","cluster_addr":"172.21.15.37:6825/3336489453","heartbeat_back_addr":"172.21.15.37:6833/3336489453","heartbeat_front_addr":"172.21.15.37:6830/3336489453","state":["exists","up"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6811","nonce":1608953722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6818","nonce":1608953722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6827","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6829","nonce":1608953722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6821","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6825","nonce":1608953722}]},"public_addr":"172.21.15.45:6811/1608953722","cluster_addr":"172.21.15.45:6818/1608953722","heartbeat_back_addr":"172.21.15.45:6829/1608953722","heartbeat_front_addr":"172.21.15.45:6825/1608953722","state":["exists","up"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6825","nonce":3719375992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6827","nonce":3719375992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6831","nonce":3719375992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6829","nonce":3719375992}]},"public_addr":"172.21.15.71:6825/3719375992","cluster_addr":"172.21.15.71:6827/3719375992","heartbeat_back_addr":"172.21.15.71:6831/3719375992","heartbeat_front_addr":"172.21.15.71:6829/3719375992","state":["exists","up"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6805","nonce":2891128021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6811","nonce":2891128021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6817","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6819","nonce":2891128021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6815","nonce":2891128021}]},"public_addr":"172.21.15.181:6805/2891128021","cluster_addr":"172.21.15.181:6811/2891128021","heartbeat_back_addr":"172.21.15.181:6819/2891128021","heartbeat_front_addr":"172.21.15.181:6815/2891128021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.005223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.094705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.055517+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.150223+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.014893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.069228+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.996686+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.925609+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.140441+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.941965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.103406+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.093266+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.998590+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.192803+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.797282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:23:49.665 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-04-26T01:23:49.665 INFO:tasks.ceph:Creating RBD pool 2024-04-26T01:23:49.666 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-26T01:23:50.886 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'rbd' created 2024-04-26T01:23:50.912 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-26T01:23:51.889 INFO:teuthology.orchestra.run.smithi037.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-26T01:23:51.920 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T01:23:51.921 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T01:23:51.921 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T01:23:51.921 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T01:23:51.922 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T01:23:52.277 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:52.295 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":4,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":777549452},{"type":"v1","addr":"172.21.15.37:6801","nonce":777549452}]},"active_addr":"172.21.15.37:6801/777549452","active_change":"2024-04-26T01:23:39.809794+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":2552870528}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":926383036}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":2027732674}]}]} 2024-04-26T01:23:52.301 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T01:23:52.301 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T01:23:52.302 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:23:52.612 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:52.612 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":26,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:23:51.891170+0000","last_up_change":"2024-04-26T01:23:47.867650+0000","last_in_change":"2024-04-26T01:23:39.528016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:23:47.920762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:23:50.025173+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6819","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6820","nonce":906916438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6824","nonce":906916438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6829","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6831","nonce":906916438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6827","nonce":906916438}]},"public_addr":"172.21.15.37:6820/906916438","cluster_addr":"172.21.15.37:6824/906916438","heartbeat_back_addr":"172.21.15.37:6831/906916438","heartbeat_front_addr":"172.21.15.37:6827/906916438","state":["exists","up"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6813","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6816","nonce":1630954663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6823","nonce":1630954663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6831","nonce":1630954663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6828","nonce":1630954663}]},"public_addr":"172.21.15.45:6816/1630954663","cluster_addr":"172.21.15.45:6823/1630954663","heartbeat_back_addr":"172.21.15.45:6831/1630954663","heartbeat_front_addr":"172.21.15.45:6828/1630954663","state":["exists","up"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6801","nonce":2961312835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6803","nonce":2961312835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6807","nonce":2961312835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6805","nonce":2961312835}]},"public_addr":"172.21.15.71:6801/2961312835","cluster_addr":"172.21.15.71:6803/2961312835","heartbeat_back_addr":"172.21.15.71:6807/2961312835","heartbeat_front_addr":"172.21.15.71:6805/2961312835","state":["exists","up"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6801","nonce":1026435117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6806","nonce":1026435117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6821","nonce":1026435117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6814","nonce":1026435117}]},"public_addr":"172.21.15.181:6801/1026435117","cluster_addr":"172.21.15.181:6806/1026435117","heartbeat_back_addr":"172.21.15.181:6821/1026435117","heartbeat_front_addr":"172.21.15.181:6814/1026435117","state":["exists","up"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6803","nonce":1575437742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6805","nonce":1575437742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6809","nonce":1575437742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6807","nonce":1575437742}]},"public_addr":"172.21.15.37:6803/1575437742","cluster_addr":"172.21.15.37:6805/1575437742","heartbeat_back_addr":"172.21.15.37:6809/1575437742","heartbeat_front_addr":"172.21.15.37:6807/1575437742","state":["exists","up"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6801","nonce":3759230996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6803","nonce":3759230996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6807","nonce":3759230996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6805","nonce":3759230996}]},"public_addr":"172.21.15.45:6801/3759230996","cluster_addr":"172.21.15.45:6803/3759230996","heartbeat_back_addr":"172.21.15.45:6807/3759230996","heartbeat_front_addr":"172.21.15.45:6805/3759230996","state":["exists","up"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6809","nonce":76180415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6811","nonce":76180415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6815","nonce":76180415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6813","nonce":76180415}]},"public_addr":"172.21.15.71:6809/76180415","cluster_addr":"172.21.15.71:6811/76180415","heartbeat_back_addr":"172.21.15.71:6815/76180415","heartbeat_front_addr":"172.21.15.71:6813/76180415","state":["exists","up"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6807","nonce":4043929361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6812","nonce":4043929361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6823","nonce":4043929361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6820","nonce":4043929361}]},"public_addr":"172.21.15.181:6807/4043929361","cluster_addr":"172.21.15.181:6812/4043929361","heartbeat_back_addr":"172.21.15.181:6823/4043929361","heartbeat_front_addr":"172.21.15.181:6820/4043929361","state":["exists","up"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6811","nonce":4094482142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6813","nonce":4094482142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6817","nonce":4094482142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6815","nonce":4094482142}]},"public_addr":"172.21.15.37:6811/4094482142","cluster_addr":"172.21.15.37:6813/4094482142","heartbeat_back_addr":"172.21.15.37:6817/4094482142","heartbeat_front_addr":"172.21.15.37:6815/4094482142","state":["exists","up"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6810","nonce":3448396888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6815","nonce":3448396888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6824","nonce":3448396888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6817","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6819","nonce":3448396888}]},"public_addr":"172.21.15.45:6810/3448396888","cluster_addr":"172.21.15.45:6815/3448396888","heartbeat_back_addr":"172.21.15.45:6824/3448396888","heartbeat_front_addr":"172.21.15.45:6819/3448396888","state":["exists","up"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6817","nonce":896289472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6819","nonce":896289472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6823","nonce":896289472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6821","nonce":896289472}]},"public_addr":"172.21.15.71:6817/896289472","cluster_addr":"172.21.15.71:6819/896289472","heartbeat_back_addr":"172.21.15.71:6823/896289472","heartbeat_front_addr":"172.21.15.71:6821/896289472","state":["exists","up"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6825","nonce":647568313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6827","nonce":647568313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6831","nonce":647568313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6829","nonce":647568313}]},"public_addr":"172.21.15.181:6825/647568313","cluster_addr":"172.21.15.181:6827/647568313","heartbeat_back_addr":"172.21.15.181:6831/647568313","heartbeat_front_addr":"172.21.15.181:6829/647568313","state":["exists","up"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6821","nonce":3336489453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6825","nonce":3336489453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6833","nonce":3336489453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6830","nonce":3336489453}]},"public_addr":"172.21.15.37:6821/3336489453","cluster_addr":"172.21.15.37:6825/3336489453","heartbeat_back_addr":"172.21.15.37:6833/3336489453","heartbeat_front_addr":"172.21.15.37:6830/3336489453","state":["exists","up"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6811","nonce":1608953722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6818","nonce":1608953722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6827","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6829","nonce":1608953722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6821","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6825","nonce":1608953722}]},"public_addr":"172.21.15.45:6811/1608953722","cluster_addr":"172.21.15.45:6818/1608953722","heartbeat_back_addr":"172.21.15.45:6829/1608953722","heartbeat_front_addr":"172.21.15.45:6825/1608953722","state":["exists","up"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6825","nonce":3719375992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6827","nonce":3719375992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6831","nonce":3719375992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6829","nonce":3719375992}]},"public_addr":"172.21.15.71:6825/3719375992","cluster_addr":"172.21.15.71:6827/3719375992","heartbeat_back_addr":"172.21.15.71:6831/3719375992","heartbeat_front_addr":"172.21.15.71:6829/3719375992","state":["exists","up"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6805","nonce":2891128021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6811","nonce":2891128021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6817","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6819","nonce":2891128021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6815","nonce":2891128021}]},"public_addr":"172.21.15.181:6805/2891128021","cluster_addr":"172.21.15.181:6811/2891128021","heartbeat_back_addr":"172.21.15.181:6819/2891128021","heartbeat_front_addr":"172.21.15.181:6815/2891128021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.005223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.094705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.055517+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.150223+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.014893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.069228+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.996686+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.925609+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.140441+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.941965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.103406+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.093266+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.998590+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.192803+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.797282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:23:52.631 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T01:23:52.631 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:23:52.945 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:52.945 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":27,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:23:52.908386+0000","last_up_change":"2024-04-26T01:23:47.867650+0000","last_in_change":"2024-04-26T01:23:39.528016+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:23:47.920762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:23:50.025173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6819","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6820","nonce":906916438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6824","nonce":906916438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6829","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6831","nonce":906916438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6827","nonce":906916438}]},"public_addr":"172.21.15.37:6820/906916438","cluster_addr":"172.21.15.37:6824/906916438","heartbeat_back_addr":"172.21.15.37:6831/906916438","heartbeat_front_addr":"172.21.15.37:6827/906916438","state":["exists","up"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6813","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6816","nonce":1630954663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6823","nonce":1630954663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6831","nonce":1630954663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6828","nonce":1630954663}]},"public_addr":"172.21.15.45:6816/1630954663","cluster_addr":"172.21.15.45:6823/1630954663","heartbeat_back_addr":"172.21.15.45:6831/1630954663","heartbeat_front_addr":"172.21.15.45:6828/1630954663","state":["exists","up"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6801","nonce":2961312835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6803","nonce":2961312835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6807","nonce":2961312835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6805","nonce":2961312835}]},"public_addr":"172.21.15.71:6801/2961312835","cluster_addr":"172.21.15.71:6803/2961312835","heartbeat_back_addr":"172.21.15.71:6807/2961312835","heartbeat_front_addr":"172.21.15.71:6805/2961312835","state":["exists","up"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6801","nonce":1026435117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6806","nonce":1026435117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6821","nonce":1026435117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6814","nonce":1026435117}]},"public_addr":"172.21.15.181:6801/1026435117","cluster_addr":"172.21.15.181:6806/1026435117","heartbeat_back_addr":"172.21.15.181:6821/1026435117","heartbeat_front_addr":"172.21.15.181:6814/1026435117","state":["exists","up"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6803","nonce":1575437742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6805","nonce":1575437742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6809","nonce":1575437742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6807","nonce":1575437742}]},"public_addr":"172.21.15.37:6803/1575437742","cluster_addr":"172.21.15.37:6805/1575437742","heartbeat_back_addr":"172.21.15.37:6809/1575437742","heartbeat_front_addr":"172.21.15.37:6807/1575437742","state":["exists","up"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6801","nonce":3759230996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6803","nonce":3759230996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6807","nonce":3759230996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6805","nonce":3759230996}]},"public_addr":"172.21.15.45:6801/3759230996","cluster_addr":"172.21.15.45:6803/3759230996","heartbeat_back_addr":"172.21.15.45:6807/3759230996","heartbeat_front_addr":"172.21.15.45:6805/3759230996","state":["exists","up"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6809","nonce":76180415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6811","nonce":76180415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6815","nonce":76180415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6813","nonce":76180415}]},"public_addr":"172.21.15.71:6809/76180415","cluster_addr":"172.21.15.71:6811/76180415","heartbeat_back_addr":"172.21.15.71:6815/76180415","heartbeat_front_addr":"172.21.15.71:6813/76180415","state":["exists","up"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6807","nonce":4043929361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6812","nonce":4043929361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6823","nonce":4043929361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6820","nonce":4043929361}]},"public_addr":"172.21.15.181:6807/4043929361","cluster_addr":"172.21.15.181:6812/4043929361","heartbeat_back_addr":"172.21.15.181:6823/4043929361","heartbeat_front_addr":"172.21.15.181:6820/4043929361","state":["exists","up"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6811","nonce":4094482142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6813","nonce":4094482142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6817","nonce":4094482142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6815","nonce":4094482142}]},"public_addr":"172.21.15.37:6811/4094482142","cluster_addr":"172.21.15.37:6813/4094482142","heartbeat_back_addr":"172.21.15.37:6817/4094482142","heartbeat_front_addr":"172.21.15.37:6815/4094482142","state":["exists","up"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6810","nonce":3448396888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6815","nonce":3448396888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6824","nonce":3448396888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6817","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6819","nonce":3448396888}]},"public_addr":"172.21.15.45:6810/3448396888","cluster_addr":"172.21.15.45:6815/3448396888","heartbeat_back_addr":"172.21.15.45:6824/3448396888","heartbeat_front_addr":"172.21.15.45:6819/3448396888","state":["exists","up"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6817","nonce":896289472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6819","nonce":896289472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6823","nonce":896289472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6821","nonce":896289472}]},"public_addr":"172.21.15.71:6817/896289472","cluster_addr":"172.21.15.71:6819/896289472","heartbeat_back_addr":"172.21.15.71:6823/896289472","heartbeat_front_addr":"172.21.15.71:6821/896289472","state":["exists","up"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6825","nonce":647568313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6827","nonce":647568313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6831","nonce":647568313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6829","nonce":647568313}]},"public_addr":"172.21.15.181:6825/647568313","cluster_addr":"172.21.15.181:6827/647568313","heartbeat_back_addr":"172.21.15.181:6831/647568313","heartbeat_front_addr":"172.21.15.181:6829/647568313","state":["exists","up"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6821","nonce":3336489453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6825","nonce":3336489453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6833","nonce":3336489453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6830","nonce":3336489453}]},"public_addr":"172.21.15.37:6821/3336489453","cluster_addr":"172.21.15.37:6825/3336489453","heartbeat_back_addr":"172.21.15.37:6833/3336489453","heartbeat_front_addr":"172.21.15.37:6830/3336489453","state":["exists","up"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6811","nonce":1608953722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6818","nonce":1608953722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6827","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6829","nonce":1608953722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6821","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6825","nonce":1608953722}]},"public_addr":"172.21.15.45:6811/1608953722","cluster_addr":"172.21.15.45:6818/1608953722","heartbeat_back_addr":"172.21.15.45:6829/1608953722","heartbeat_front_addr":"172.21.15.45:6825/1608953722","state":["exists","up"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6825","nonce":3719375992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6827","nonce":3719375992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6831","nonce":3719375992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6829","nonce":3719375992}]},"public_addr":"172.21.15.71:6825/3719375992","cluster_addr":"172.21.15.71:6827/3719375992","heartbeat_back_addr":"172.21.15.71:6831/3719375992","heartbeat_front_addr":"172.21.15.71:6829/3719375992","state":["exists","up"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6805","nonce":2891128021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6811","nonce":2891128021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6817","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6819","nonce":2891128021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6815","nonce":2891128021}]},"public_addr":"172.21.15.181:6805/2891128021","cluster_addr":"172.21.15.181:6811/2891128021","heartbeat_back_addr":"172.21.15.181:6819/2891128021","heartbeat_front_addr":"172.21.15.181:6815/2891128021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.005223+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.094705+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.055517+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.150223+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.863756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.014893+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.069228+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.996686+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.925609+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.140441+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.941965+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.103406+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.093266+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.998590+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:45.192803+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:23:44.797282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:23:52.964 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T01:23:52.964 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T01:23:52.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T01:23:52.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T01:23:52.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T01:23:52.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T01:23:52.966 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-26T01:23:52.967 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-26T01:23:52.967 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-26T01:23:52.968 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-26T01:23:52.968 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-26T01:23:52.968 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-26T01:23:52.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-04-26T01:23:52.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-04-26T01:23:52.969 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-04-26T01:23:52.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-04-26T01:23:53.392 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:53.393 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-26T01:23:53.448 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-26T01:23:53.450 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-04-26T01:23:53.450 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T01:23:53.460 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.461 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T01:23:53.462 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.463 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T01:23:53.467 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.468 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T01:23:53.485 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:53.486 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-26T01:23:53.497 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:53.497 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T01:23:53.507 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.507 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T01:23:53.512 INFO:teuthology.orchestra.run.smithi037.stdout:94489280514 2024-04-26T01:23:53.512 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-26T01:23:53.515 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.516 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-26T01:23:53.522 INFO:teuthology.orchestra.run.smithi037.stdout:90194313221 2024-04-26T01:23:53.522 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-26T01:23:53.526 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:53.526 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T01:23:53.538 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.538 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T01:23:53.541 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-04-26T01:23:53.542 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-26T01:23:53.582 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:53.583 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-26T01:23:54.222 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:54.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.10 2024-04-26T01:23:54.279 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.304 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:54.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.4 2024-04-26T01:23:54.365 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.439 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:54.454 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.463 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.495 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.496 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.15 2024-04-26T01:23:54.497 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.503 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.13 2024-04-26T01:23:54.504 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.533 INFO:teuthology.orchestra.run.smithi037.stdout:90194313219 2024-04-26T01:23:54.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.2 2024-04-26T01:23:54.539 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.548 INFO:teuthology.orchestra.run.smithi037.stdout:90194313221 2024-04-26T01:23:54.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.3 2024-04-26T01:23:54.551 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.560 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.572 INFO:teuthology.orchestra.run.smithi037.stdout:94489280514 2024-04-26T01:23:54.582 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.1 2024-04-26T01:23:54.582 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.582 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.583 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-04-26T01:23:54.594 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.11 2024-04-26T01:23:54.594 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.607 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.5 2024-04-26T01:23:54.608 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280514 got 94489280514 for osd.8 2024-04-26T01:23:54.614 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.621 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.14 2024-04-26T01:23:54.621 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.622 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2024-04-26T01:23:54.622 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.628 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.635 INFO:teuthology.orchestra.run.smithi037.stdout:94489280516 2024-04-26T01:23:54.643 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.650 INFO:teuthology.orchestra.run.smithi037.stdout:90194313220 2024-04-26T01:23:54.656 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.12 2024-04-26T01:23:54.656 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.6 2024-04-26T01:23:54.658 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.9 2024-04-26T01:23:54.665 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.671 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.7 2024-04-26T01:23:54.672 DEBUG:teuthology.parallel:result is None 2024-04-26T01:23:54.672 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T01:23:54.672 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:23:54.995 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:54.995 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:23:55.011 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-26T01:23:53.818096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444700,"kb_used_data":3956,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504740,"statfs":{"total":1546188226560,"available":1545732853760,"internally_reserved":0,"allocated":4050944,"data_stored":1956270,"data_compressed":195183,"data_compressed_allocated":282624,"data_compressed_original":794624,"omap_allocated":25521,"internal_metadata":450993231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.332504+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:23:53.332504+0000","last_peered":"2024-04-26T01:23:53.332504+0000","last_clean":"2024-04-26T01:23:53.332504+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:23:53.332504+0000","last_undegraded":"2024-04-26T01:23:53.332504+0000","last_fullsized":"2024-04-26T01:23:53.332504+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.016347+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:23:52.016347+0000","last_peered":"2024-04-26T01:23:52.016347+0000","last_clean":"2024-04-26T01:23:52.016347+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:23:52.016347+0000","last_undegraded":"2024-04-26T01:23:52.016347+0000","last_fullsized":"2024-04-26T01:23:52.016347+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:54.066666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329576+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:23:53.329576+0000","last_peered":"2024-04-26T01:23:53.329576+0000","last_clean":"2024-04-26T01:23:53.329576+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:23:53.329576+0000","last_undegraded":"2024-04-26T01:23:53.329576+0000","last_fullsized":"2024-04-26T01:23:53.329576+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.919637+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:23:52.919637+0000","last_peered":"2024-04-26T01:23:52.919637+0000","last_clean":"2024-04-26T01:23:52.919637+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:23:52.919637+0000","last_undegraded":"2024-04-26T01:23:52.919637+0000","last_fullsized":"2024-04-26T01:23:52.919637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.014687+0000","last_change":"2024-04-26T01:23:52.014687+0000","last_active":"2024-04-26T01:23:52.014687+0000","last_peered":"2024-04-26T01:23:52.014687+0000","last_clean":"2024-04-26T01:23:52.014687+0000","last_became_active":"2024-04-26T01:23:52.014133+0000","last_became_peered":"2024-04-26T01:23:52.014133+0000","last_unstale":"2024-04-26T01:23:52.014687+0000","last_undegraded":"2024-04-26T01:23:52.014687+0000","last_fullsized":"2024-04-26T01:23:52.014687+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329564+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:23:53.329564+0000","last_peered":"2024-04-26T01:23:53.329564+0000","last_clean":"2024-04-26T01:23:53.329564+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:23:53.329564+0000","last_undegraded":"2024-04-26T01:23:53.329564+0000","last_fullsized":"2024-04-26T01:23:53.329564+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.211699+0000","last_change":"2024-04-26T01:23:52.211699+0000","last_active":"2024-04-26T01:23:52.211699+0000","last_peered":"2024-04-26T01:23:52.211699+0000","last_clean":"2024-04-26T01:23:52.211699+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:23:52.211699+0000","last_undegraded":"2024-04-26T01:23:52.211699+0000","last_fullsized":"2024-04-26T01:23:52.211699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.071063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.021337+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:23:53.021337+0000","last_peered":"2024-04-26T01:23:53.021337+0000","last_clean":"2024-04-26T01:23:53.021337+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:23:53.021337+0000","last_undegraded":"2024-04-26T01:23:53.021337+0000","last_fullsized":"2024-04-26T01:23:53.021337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'32","reported_seq":62,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.369961+0000","last_change":"2024-04-26T01:23:49.898546+0000","last_active":"2024-04-26T01:23:53.369961+0000","last_peered":"2024-04-26T01:23:53.369961+0000","last_clean":"2024-04-26T01:23:53.369961+0000","last_became_active":"2024-04-26T01:23:49.896731+0000","last_became_peered":"2024-04-26T01:23:49.896731+0000","last_unstale":"2024-04-26T01:23:53.369961+0000","last_undegraded":"2024-04-26T01:23:53.369961+0000","last_fullsized":"2024-04-26T01:23:53.369961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:23:48.870422+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:04.173495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12730,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113728,"data_compressed":14048,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:23:55.012 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:23:55.292 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:55.293 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:23:55.309 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-04-26T01:23:53.818096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444700,"kb_used_data":3956,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504740,"statfs":{"total":1546188226560,"available":1545732853760,"internally_reserved":0,"allocated":4050944,"data_stored":1956270,"data_compressed":195183,"data_compressed_allocated":282624,"data_compressed_original":794624,"omap_allocated":25521,"internal_metadata":450993231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.332504+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:23:53.332504+0000","last_peered":"2024-04-26T01:23:53.332504+0000","last_clean":"2024-04-26T01:23:53.332504+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:23:53.332504+0000","last_undegraded":"2024-04-26T01:23:53.332504+0000","last_fullsized":"2024-04-26T01:23:53.332504+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.016347+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:23:52.016347+0000","last_peered":"2024-04-26T01:23:52.016347+0000","last_clean":"2024-04-26T01:23:52.016347+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:23:52.016347+0000","last_undegraded":"2024-04-26T01:23:52.016347+0000","last_fullsized":"2024-04-26T01:23:52.016347+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:54.066666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329576+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:23:53.329576+0000","last_peered":"2024-04-26T01:23:53.329576+0000","last_clean":"2024-04-26T01:23:53.329576+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:23:53.329576+0000","last_undegraded":"2024-04-26T01:23:53.329576+0000","last_fullsized":"2024-04-26T01:23:53.329576+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.919637+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:23:52.919637+0000","last_peered":"2024-04-26T01:23:52.919637+0000","last_clean":"2024-04-26T01:23:52.919637+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:23:52.919637+0000","last_undegraded":"2024-04-26T01:23:52.919637+0000","last_fullsized":"2024-04-26T01:23:52.919637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.014687+0000","last_change":"2024-04-26T01:23:52.014687+0000","last_active":"2024-04-26T01:23:52.014687+0000","last_peered":"2024-04-26T01:23:52.014687+0000","last_clean":"2024-04-26T01:23:52.014687+0000","last_became_active":"2024-04-26T01:23:52.014133+0000","last_became_peered":"2024-04-26T01:23:52.014133+0000","last_unstale":"2024-04-26T01:23:52.014687+0000","last_undegraded":"2024-04-26T01:23:52.014687+0000","last_fullsized":"2024-04-26T01:23:52.014687+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329564+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:23:53.329564+0000","last_peered":"2024-04-26T01:23:53.329564+0000","last_clean":"2024-04-26T01:23:53.329564+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:23:53.329564+0000","last_undegraded":"2024-04-26T01:23:53.329564+0000","last_fullsized":"2024-04-26T01:23:53.329564+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.211699+0000","last_change":"2024-04-26T01:23:52.211699+0000","last_active":"2024-04-26T01:23:52.211699+0000","last_peered":"2024-04-26T01:23:52.211699+0000","last_clean":"2024-04-26T01:23:52.211699+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:23:52.211699+0000","last_undegraded":"2024-04-26T01:23:52.211699+0000","last_fullsized":"2024-04-26T01:23:52.211699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.071063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.021337+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:23:53.021337+0000","last_peered":"2024-04-26T01:23:53.021337+0000","last_clean":"2024-04-26T01:23:53.021337+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:23:53.021337+0000","last_undegraded":"2024-04-26T01:23:53.021337+0000","last_fullsized":"2024-04-26T01:23:53.021337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'32","reported_seq":62,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.369961+0000","last_change":"2024-04-26T01:23:49.898546+0000","last_active":"2024-04-26T01:23:53.369961+0000","last_peered":"2024-04-26T01:23:53.369961+0000","last_clean":"2024-04-26T01:23:53.369961+0000","last_became_active":"2024-04-26T01:23:49.896731+0000","last_became_peered":"2024-04-26T01:23:49.896731+0000","last_unstale":"2024-04-26T01:23:53.369961+0000","last_undegraded":"2024-04-26T01:23:53.369961+0000","last_fullsized":"2024-04-26T01:23:53.369961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:23:48.870422+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:04.173495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12730,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113728,"data_compressed":14048,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:23:55.310 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T01:23:55.310 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T01:23:55.310 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T01:23:55.310 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T01:23:55.619 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:55.619 INFO:teuthology.orchestra.run.smithi037.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T01:23:55.636 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T01:23:55.637 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-26T01:23:55.648 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-26T01:23:55.648 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b22e2ebdeb24376882b7bda2a7329c8cccc2276a'}} 2024-04-26T01:23:55.648 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-26T01:23:55.648 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-26T01:23:55.648 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T01:23:55.649 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:23:55.933 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:55.933 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:23:55.949 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-26T01:23:55.818570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444700,"kb_used_data":3956,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504740,"statfs":{"total":1546188226560,"available":1545732853760,"internally_reserved":0,"allocated":4050944,"data_stored":1956270,"data_compressed":195183,"data_compressed_allocated":282624,"data_compressed_original":794624,"omap_allocated":25521,"internal_metadata":450993231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000474"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.332504+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:23:53.332504+0000","last_peered":"2024-04-26T01:23:53.332504+0000","last_clean":"2024-04-26T01:23:53.332504+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:23:53.332504+0000","last_undegraded":"2024-04-26T01:23:53.332504+0000","last_fullsized":"2024-04-26T01:23:53.332504+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.016347+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:23:52.016347+0000","last_peered":"2024-04-26T01:23:52.016347+0000","last_clean":"2024-04-26T01:23:52.016347+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:23:52.016347+0000","last_undegraded":"2024-04-26T01:23:52.016347+0000","last_fullsized":"2024-04-26T01:23:52.016347+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:54.066666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329576+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:23:53.329576+0000","last_peered":"2024-04-26T01:23:53.329576+0000","last_clean":"2024-04-26T01:23:53.329576+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:23:53.329576+0000","last_undegraded":"2024-04-26T01:23:53.329576+0000","last_fullsized":"2024-04-26T01:23:53.329576+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.919637+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:23:52.919637+0000","last_peered":"2024-04-26T01:23:52.919637+0000","last_clean":"2024-04-26T01:23:52.919637+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:23:52.919637+0000","last_undegraded":"2024-04-26T01:23:52.919637+0000","last_fullsized":"2024-04-26T01:23:52.919637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.014687+0000","last_change":"2024-04-26T01:23:52.014687+0000","last_active":"2024-04-26T01:23:52.014687+0000","last_peered":"2024-04-26T01:23:52.014687+0000","last_clean":"2024-04-26T01:23:52.014687+0000","last_became_active":"2024-04-26T01:23:52.014133+0000","last_became_peered":"2024-04-26T01:23:52.014133+0000","last_unstale":"2024-04-26T01:23:52.014687+0000","last_undegraded":"2024-04-26T01:23:52.014687+0000","last_fullsized":"2024-04-26T01:23:52.014687+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329564+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:23:53.329564+0000","last_peered":"2024-04-26T01:23:53.329564+0000","last_clean":"2024-04-26T01:23:53.329564+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:23:53.329564+0000","last_undegraded":"2024-04-26T01:23:53.329564+0000","last_fullsized":"2024-04-26T01:23:53.329564+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.211699+0000","last_change":"2024-04-26T01:23:52.211699+0000","last_active":"2024-04-26T01:23:52.211699+0000","last_peered":"2024-04-26T01:23:52.211699+0000","last_clean":"2024-04-26T01:23:52.211699+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:23:52.211699+0000","last_undegraded":"2024-04-26T01:23:52.211699+0000","last_fullsized":"2024-04-26T01:23:52.211699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.071063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.021337+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:23:53.021337+0000","last_peered":"2024-04-26T01:23:53.021337+0000","last_clean":"2024-04-26T01:23:53.021337+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:23:53.021337+0000","last_undegraded":"2024-04-26T01:23:53.021337+0000","last_fullsized":"2024-04-26T01:23:53.021337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'32","reported_seq":62,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.369961+0000","last_change":"2024-04-26T01:23:49.898546+0000","last_active":"2024-04-26T01:23:53.369961+0000","last_peered":"2024-04-26T01:23:53.369961+0000","last_clean":"2024-04-26T01:23:53.369961+0000","last_became_active":"2024-04-26T01:23:49.896731+0000","last_became_peered":"2024-04-26T01:23:49.896731+0000","last_unstale":"2024-04-26T01:23:53.369961+0000","last_undegraded":"2024-04-26T01:23:53.369961+0000","last_fullsized":"2024-04-26T01:23:53.369961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:23:48.870422+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:04.173495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12730,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113728,"data_compressed":14048,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:23:55.950 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:23:56.245 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:23:56.245 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:23:56.262 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-26T01:23:55.818570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":444700,"kb_used_data":3956,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504740,"statfs":{"total":1546188226560,"available":1545732853760,"internally_reserved":0,"allocated":4050944,"data_stored":1956270,"data_compressed":195183,"data_compressed_allocated":282624,"data_compressed_original":794624,"omap_allocated":25521,"internal_metadata":450993231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000474"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.332504+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:23:53.332504+0000","last_peered":"2024-04-26T01:23:53.332504+0000","last_clean":"2024-04-26T01:23:53.332504+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:23:53.332504+0000","last_undegraded":"2024-04-26T01:23:53.332504+0000","last_fullsized":"2024-04-26T01:23:53.332504+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.016347+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:23:52.016347+0000","last_peered":"2024-04-26T01:23:52.016347+0000","last_clean":"2024-04-26T01:23:52.016347+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:23:52.016347+0000","last_undegraded":"2024-04-26T01:23:52.016347+0000","last_fullsized":"2024-04-26T01:23:52.016347+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:54.066666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329576+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:23:53.329576+0000","last_peered":"2024-04-26T01:23:53.329576+0000","last_clean":"2024-04-26T01:23:53.329576+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:23:53.329576+0000","last_undegraded":"2024-04-26T01:23:53.329576+0000","last_fullsized":"2024-04-26T01:23:53.329576+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.919637+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:23:52.919637+0000","last_peered":"2024-04-26T01:23:52.919637+0000","last_clean":"2024-04-26T01:23:52.919637+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:23:52.919637+0000","last_undegraded":"2024-04-26T01:23:52.919637+0000","last_fullsized":"2024-04-26T01:23:52.919637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.014687+0000","last_change":"2024-04-26T01:23:52.014687+0000","last_active":"2024-04-26T01:23:52.014687+0000","last_peered":"2024-04-26T01:23:52.014687+0000","last_clean":"2024-04-26T01:23:52.014687+0000","last_became_active":"2024-04-26T01:23:52.014133+0000","last_became_peered":"2024-04-26T01:23:52.014133+0000","last_unstale":"2024-04-26T01:23:52.014687+0000","last_undegraded":"2024-04-26T01:23:52.014687+0000","last_fullsized":"2024-04-26T01:23:52.014687+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.329564+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:23:53.329564+0000","last_peered":"2024-04-26T01:23:53.329564+0000","last_clean":"2024-04-26T01:23:53.329564+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:23:53.329564+0000","last_undegraded":"2024-04-26T01:23:53.329564+0000","last_fullsized":"2024-04-26T01:23:53.329564+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-26T01:23:52.211699+0000","last_change":"2024-04-26T01:23:52.211699+0000","last_active":"2024-04-26T01:23:52.211699+0000","last_peered":"2024-04-26T01:23:52.211699+0000","last_clean":"2024-04-26T01:23:52.211699+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:23:52.211699+0000","last_undegraded":"2024-04-26T01:23:52.211699+0000","last_fullsized":"2024-04-26T01:23:52.211699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.071063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.021337+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:23:53.021337+0000","last_peered":"2024-04-26T01:23:53.021337+0000","last_clean":"2024-04-26T01:23:53.021337+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:23:53.021337+0000","last_undegraded":"2024-04-26T01:23:53.021337+0000","last_fullsized":"2024-04-26T01:23:53.021337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"27'32","reported_seq":62,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-26T01:23:53.369961+0000","last_change":"2024-04-26T01:23:49.898546+0000","last_active":"2024-04-26T01:23:53.369961+0000","last_peered":"2024-04-26T01:23:53.369961+0000","last_clean":"2024-04-26T01:23:53.369961+0000","last_became_active":"2024-04-26T01:23:49.896731+0000","last_became_peered":"2024-04-26T01:23:49.896731+0000","last_unstale":"2024-04-26T01:23:53.369961+0000","last_undegraded":"2024-04-26T01:23:53.369961+0000","last_fullsized":"2024-04-26T01:23:53.369961+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:23:48.870422+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:04.173495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":262672,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":58,"num_write_kb":394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":278552,"data_compressed":12730,"data_compressed_allocated":16384,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,10,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27268,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27288,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":262144,"data_stored":113728,"data_compressed":14048,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7,9,10,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,3,4,5,6,7,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31428,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340412,"statfs":{"total":96636764160,"available":96604581888,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313220,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":104266,"data_compressed":11227,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,3,4,5,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313220,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":270336,"data_stored":243542,"data_compressed":17592,"data_compressed_allocated":24576,"data_compressed_original":163840,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":139276,"data_compressed":6365,"data_compressed_allocated":8192,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:23:56.263 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T01:23:56.263 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-26T01:23:56.545 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 27 2024-04-26T01:23:56.545 INFO:teuthology.orchestra.run.smithi037.stdout:fsid ec0903d9-9475-4d3a-af50-1dbb367f53fb 2024-04-26T01:23:56.545 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-04-26T01:23:35.072358+0000 2024-04-26T01:23:56.545 INFO:teuthology.orchestra.run.smithi037.stdout:modified 2024-04-26T01:23:52.908386+0000 2024-04-26T01:23:56.545 INFO:teuthology.orchestra.run.smithi037.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T01:23:56.545 INFO:teuthology.orchestra.run.smithi037.stdout:crush_version 4 2024-04-26T01:23:56.545 INFO:teuthology.orchestra.run.smithi037.stdout:full_ratio 0.95 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:backfillfull_ratio 0.9 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:nearfull_ratio 0.85 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:require_min_compat_client luminous 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:min_compat_client jewel 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:require_osd_release squid 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:stretch_mode_enabled false 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 27 flags hashpspool stripe_width 0 application rbd read_balance_score 4.02 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:max_osd 16 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6819/906916438,v1:172.21.15.37:6820/906916438] [v2:172.21.15.37:6822/906916438,v1:172.21.15.37:6824/906916438] exists,up a5592650-ee45-45a1-94c4-487000bdb588 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6813/1630954663,v1:172.21.15.45:6816/1630954663] [v2:172.21.15.45:6820/1630954663,v1:172.21.15.45:6823/1630954663] exists,up 26af5691-fc53-4b20-9c35-012d299e978b 2024-04-26T01:23:56.546 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6800/2961312835,v1:172.21.15.71:6801/2961312835] [v2:172.21.15.71:6802/2961312835,v1:172.21.15.71:6803/2961312835] exists,up ba3d860b-252b-42de-a951-99329413b1bf 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/1026435117,v1:172.21.15.181:6801/1026435117] [v2:172.21.15.181:6804/1026435117,v1:172.21.15.181:6806/1026435117] exists,up 8294850e-c851-4be8-adb0-ffe9ef4fd827 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6802/1575437742,v1:172.21.15.37:6803/1575437742] [v2:172.21.15.37:6804/1575437742,v1:172.21.15.37:6805/1575437742] exists,up 601a444c-5cc7-4ae1-9cd3-379b6d7155da 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6800/3759230996,v1:172.21.15.45:6801/3759230996] [v2:172.21.15.45:6802/3759230996,v1:172.21.15.45:6803/3759230996] exists,up 6818ea43-206d-49df-9df8-53d9ff132f8e 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6808/76180415,v1:172.21.15.71:6809/76180415] [v2:172.21.15.71:6810/76180415,v1:172.21.15.71:6811/76180415] exists,up 7e47d5e9-00aa-428f-8b6c-256af34a821c 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6803/4043929361,v1:172.21.15.181:6807/4043929361] [v2:172.21.15.181:6809/4043929361,v1:172.21.15.181:6812/4043929361] exists,up 0ed4ac32-3700-4879-abde-e664c307dcdf 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6810/4094482142,v1:172.21.15.37:6811/4094482142] [v2:172.21.15.37:6812/4094482142,v1:172.21.15.37:6813/4094482142] exists,up 94604649-8873-4348-a27c-47e6751274c4 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6808/3448396888,v1:172.21.15.45:6810/3448396888] [v2:172.21.15.45:6812/3448396888,v1:172.21.15.45:6815/3448396888] exists,up cf4159cb-22fe-49c9-b94c-36a6003e6c3b 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6816/896289472,v1:172.21.15.71:6817/896289472] [v2:172.21.15.71:6818/896289472,v1:172.21.15.71:6819/896289472] exists,up bf42aad2-b60d-45a4-adde-f258ce7086b3 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/647568313,v1:172.21.15.181:6825/647568313] [v2:172.21.15.181:6826/647568313,v1:172.21.15.181:6827/647568313] exists,up 1db3483a-2bc5-4cb5-9034-224068e68cdd 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6818/3336489453,v1:172.21.15.37:6821/3336489453] [v2:172.21.15.37:6823/3336489453,v1:172.21.15.37:6825/3336489453] exists,up 970dc59e-69c3-4a15-ada5-569b3ee520b6 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6809/1608953722,v1:172.21.15.45:6811/1608953722] [v2:172.21.15.45:6814/1608953722,v1:172.21.15.45:6818/1608953722] exists,up cae6ef09-e4e5-4d6c-8934-065720075a50 2024-04-26T01:23:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6824/3719375992,v1:172.21.15.71:6825/3719375992] [v2:172.21.15.71:6826/3719375992,v1:172.21.15.71:6827/3719375992] exists,up 02493b55-1966-4731-8bba-891cfb8f2244 2024-04-26T01:23:56.548 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6802/2891128021,v1:172.21.15.181:6805/2891128021] [v2:172.21.15.181:6808/2891128021,v1:172.21.15.181:6811/2891128021] exists,up 93d2b85d-c322-44b6-aa6a-c5da975ba45a 2024-04-26T01:23:56.562 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6819/906916438,v1:172.21.15.37:6820/906916438] [v2:172.21.15.37:6822/906916438,v1:172.21.15.37:6824/906916438] exists,up a5592650-ee45-45a1-94c4-487000bdb588', 'osd.1 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6813/1630954663,v1:172.21.15.45:6816/1630954663] [v2:172.21.15.45:6820/1630954663,v1:172.21.15.45:6823/1630954663] exists,up 26af5691-fc53-4b20-9c35-012d299e978b', 'osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6800/2961312835,v1:172.21.15.71:6801/2961312835] [v2:172.21.15.71:6802/2961312835,v1:172.21.15.71:6803/2961312835] exists,up ba3d860b-252b-42de-a951-99329413b1bf', 'osd.3 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/1026435117,v1:172.21.15.181:6801/1026435117] [v2:172.21.15.181:6804/1026435117,v1:172.21.15.181:6806/1026435117] exists,up 8294850e-c851-4be8-adb0-ffe9ef4fd827', 'osd.4 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6802/1575437742,v1:172.21.15.37:6803/1575437742] [v2:172.21.15.37:6804/1575437742,v1:172.21.15.37:6805/1575437742] exists,up 601a444c-5cc7-4ae1-9cd3-379b6d7155da', 'osd.5 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6800/3759230996,v1:172.21.15.45:6801/3759230996] [v2:172.21.15.45:6802/3759230996,v1:172.21.15.45:6803/3759230996] exists,up 6818ea43-206d-49df-9df8-53d9ff132f8e', 'osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6808/76180415,v1:172.21.15.71:6809/76180415] [v2:172.21.15.71:6810/76180415,v1:172.21.15.71:6811/76180415] exists,up 7e47d5e9-00aa-428f-8b6c-256af34a821c', 'osd.7 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6803/4043929361,v1:172.21.15.181:6807/4043929361] [v2:172.21.15.181:6809/4043929361,v1:172.21.15.181:6812/4043929361] exists,up 0ed4ac32-3700-4879-abde-e664c307dcdf', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6810/4094482142,v1:172.21.15.37:6811/4094482142] [v2:172.21.15.37:6812/4094482142,v1:172.21.15.37:6813/4094482142] exists,up 94604649-8873-4348-a27c-47e6751274c4', 'osd.9 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6808/3448396888,v1:172.21.15.45:6810/3448396888] [v2:172.21.15.45:6812/3448396888,v1:172.21.15.45:6815/3448396888] exists,up cf4159cb-22fe-49c9-b94c-36a6003e6c3b', 'osd.10 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6816/896289472,v1:172.21.15.71:6817/896289472] [v2:172.21.15.71:6818/896289472,v1:172.21.15.71:6819/896289472] exists,up bf42aad2-b60d-45a4-adde-f258ce7086b3', 'osd.11 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/647568313,v1:172.21.15.181:6825/647568313] [v2:172.21.15.181:6826/647568313,v1:172.21.15.181:6827/647568313] exists,up 1db3483a-2bc5-4cb5-9034-224068e68cdd', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.37:6818/3336489453,v1:172.21.15.37:6821/3336489453] [v2:172.21.15.37:6823/3336489453,v1:172.21.15.37:6825/3336489453] exists,up 970dc59e-69c3-4a15-ada5-569b3ee520b6', 'osd.13 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.45:6809/1608953722,v1:172.21.15.45:6811/1608953722] [v2:172.21.15.45:6814/1608953722,v1:172.21.15.45:6818/1608953722] exists,up cae6ef09-e4e5-4d6c-8934-065720075a50', 'osd.14 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6824/3719375992,v1:172.21.15.71:6825/3719375992] [v2:172.21.15.71:6826/3719375992,v1:172.21.15.71:6827/3719375992] exists,up 02493b55-1966-4731-8bba-891cfb8f2244', 'osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6802/2891128021,v1:172.21.15.181:6805/2891128021] [v2:172.21.15.181:6808/2891128021,v1:172.21.15.181:6811/2891128021] exists,up 93d2b85d-c322-44b6-aa6a-c5da975ba45a'] 2024-04-26T01:23:56.563 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-26T01:23:56.670 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:23:56.670 INFO:teuthology.orchestra.run.smithi037.stdout: "name": "mon.a", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster": "ceph", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "admin_socket_mode": "", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_client_required": "cephx, none", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_cluster_required": "cephx", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_debug": "true", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_service_required": "cephx", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "auth_supported": "cephx", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio": "true", 2024-04-26T01:23:56.671 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio_poll_ms": "250", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_aio_reap_max": "16", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_async_discard": "false", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_block_size": "4096", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_aio": "false", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_debug_inflight_ios": "false", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_enable_discard": "false", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_flock_retry": "3", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_inject_crash": "0", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_ioring": "false", 2024-04-26T01:23:56.672 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_ioring_hipri": "false", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bdev_type": "", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_alloc_size": "1048576", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_allocator": "hybrid", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_buffered_io": "true", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_check_for_zeros": "false", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_compact_log_sync": "false", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-26T01:23:56.673 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_max_log_runway": "4194304", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_max_prefetch": "1048576", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_min_flush_size": "524288", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_min_log_runway": "1048576", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_replay_recovery": "false", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluefs_sync_write": "false", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_allocation_from_file": "true", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_allocator": "hybrid", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-26T01:23:56.674 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_create": "true", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_db_create": "false", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_db_path": "", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_db_size": "0", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_path": "", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_preallocate_file": "false", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_size": "107374182400", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_wal_create": "false", 2024-04-26T01:23:56.675 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_wal_path": "", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_block_wal_size": "100663296", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs": "true", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_autotune": "true", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-26T01:23:56.676 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_size": "0", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cache_type": "2q", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_clone_cow": "true", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-26T01:23:56.677 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_mode": "none", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_csum_type": "crc32c", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_freelist": "false", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-26T01:23:56.678 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_small_allocations": "0", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_default_buffered_read": "true", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_default_buffered_write": "false", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-26T01:23:56.679 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fail_eio": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mount": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_umount": "false", 2024-04-26T01:23:56.680 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_ignore_data_csum": "false", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_log_op_age": "5.000000", 2024-04-26T01:23:56.681 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_alloc_size": "0", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_blob_size": "0", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_max_deferred_txc": "32", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_min_alloc_size": "0", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_nid_prealloc": "1024", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-26T01:23:56.682 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_retry_disk_reads": "3", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_cf": "true", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_spdk_mem": "512", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-26T01:23:56.683 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_tracing": "false", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "bluestore_zero_block_detection": "false", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-26T01:23:56.684 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_cluster_require_signatures": "false", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_cluster_require_version": "2", 2024-04-26T01:23:56.685 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_require_signatures": "false", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_require_version": "2", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_service_require_signatures": "false", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_service_require_version": "2", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "cephx_sign_messages": "true", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "chdir": "", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "client_acl_type": "", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "client_asio_thread_count": "2", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "client_cache_mid": "0.750000", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "client_cache_size": "16384", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "client_caps_release_delay": "5", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "client_check_pool_perm": "true", 2024-04-26T01:23:56.686 INFO:teuthology.orchestra.run.smithi037.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_debug_force_sync_read": "false", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_debug_getattr_caps": "false", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_debug_inject_tick_delay": "0", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_die_on_failed_remount": "false", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_dirsize_rbytes": "true", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_force_lazyio": "false", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_fs": "", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_inject_release_failure": "false", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_max_inline_size": "4096", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mds_namespace": "", 2024-04-26T01:23:56.687 INFO:teuthology.orchestra.run.smithi037.stdout: "client_metadata": "", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mount_gid": "-1", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mount_timeout": "300", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mount_uid": "-1", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_mountpoint": "/", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_notify_timeout": "10", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc": "true", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_max_dirty": "104857600", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_max_objects": "1000", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_size": "209715200", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_oc_target_dirty": "8388608", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_permissions": "true", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_quota": "true", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_quota_df": "true", 2024-04-26T01:23:56.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_readahead_max_bytes": "0", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_readahead_max_periods": "4", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_readahead_min": "131072", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_reconnect_stale": "false", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_shutdown_timeout": "30", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_snapdir": ".snap", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_tick_interval": "1", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_trace": "", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_try_dentry_invalidate": "false", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_use_faked_inos": "false", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "client_use_random_mds": "false", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_graylog": "false", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_graylog_port": "12201", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_monitors": "default=true", 2024-04-26T01:23:56.689 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_syslog": "false", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "clog_to_syslog_level": "info", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_addr": "-", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_network": "", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "cluster_network_interface": "", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zlib_isal": "false", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zlib_level": "5", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zlib_winsize": "-15", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "compressor_zstd_level": "1", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_alien_op_num_threads": "6", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-26T01:23:56.690 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "crimson_seastar_cpu_cores": "", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location": "", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location_hook": "", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "crush_location_hook_timeout": "10", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "daemonize": "false", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "daos_pool": "tank", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_alienstore": "0/5", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_allow_any_pool_priority": "false", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_asok": "1/5", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_asok_assert_abort": "false", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_asserts_on_shutdown": "true", 2024-04-26T01:23:56.691 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_auth": "20/20", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_bdev": "1/3", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_bluefs": "1/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_bluestore": "1/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_buffer": "0/1", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_ceph_exporter": "1/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_cephfs_mirror": "0/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_cephsqlite": "0/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_client": "0/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_compressor": "1/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_context": "0/1", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_crimson_interrupt": "0/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_crush": "1/1", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_crypto": "1/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_cyanstore": "0/5", 2024-04-26T01:23:56.692 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_deliberately_leak_memory": "false", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_disable_randomized_ping": "false", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_dpdk": "1/5", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_eventtrace": "1/5", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_filer": "0/1", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_filestore": "1/3", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_finisher": "1/1", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_fuse": "1/5", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_heartbeat_testing_span": "0", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_heartbeatmap": "1/5", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_javaclient": "1/5", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_journal": "1/3", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_journaler": "0/5", 2024-04-26T01:23:56.693 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_kstore": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_lockdep": "0/1", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mclock": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_balancer": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_locker": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_log": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_log_expire": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_migrator": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mds_quiesce": "3/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_memstore": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mgr": "2/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mgrc": "1/5", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_mon": "20/20", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_monc": "0/10", 2024-04-26T01:23:56.694 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_ms": "1/1", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_none": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_objclass": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_objectcacher": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_objecter": "0/1", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_optracker": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_osd": "1/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_paxos": "20/20", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_perfcounter": "1/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_prioritycache": "1/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rados": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd_mirror": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd_pwl": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rbd_replay": "0/5", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_refs": "0/0", 2024-04-26T01:23:56.695 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_reserver": "1/1", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw": "1/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_access": "1/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_datacache": "1/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_dbstore": "1/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_flight": "1/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rgw_sync": "1/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_rocksdb": "4/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore": "0/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_backref": "0/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_cache": "0/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_cleaner": "0/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_device": "0/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_epm": "0/5", 2024-04-26T01:23:56.696 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_journal": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_lba": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_odata": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_omap": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_onode": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_t": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_seastore_tm": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_striper": "0/1", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_test": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_throttle": "1/1", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_timer": "0/1", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_tp": "0/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "debug_trace": "1/5", 2024-04-26T01:23:56.697 INFO:teuthology.orchestra.run.smithi037.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "device_failure_prediction_mode": "none", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_graylog": "false", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_journald": "false", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_stderr": "true", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "err_to_syslog": "false", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event_tracing": "false", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_addr": "0.0.0.0", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_http_port": "9926", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_prio_limit": "5", 2024-04-26T01:23:56.698 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_sort_metrics": "true", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "exporter_stats_period": "5", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "fake_statfs_for_testing": "0", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "fatal_signal_handlers": "true", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filer_max_purge_ops": "10", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filer_max_truncate_ops": "128", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_apply_finisher_threads": "1", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_blackhole": "false", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_btrfs_clone_range": "true", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_btrfs_snap": "true", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_caller_concurrency": "10", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_collect_device_partition_information": "true", 2024-04-26T01:23:56.699 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_commit_timeout": "600.000000", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_inject_read_err": "false", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_omap_check": "false", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_random_read_err": "0.000000", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_debug_verify_split": "false", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_dump_file": "", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fadvise": "true", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fail_eio": "true", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fd_cache_shards": "16", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fd_cache_size": "128", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fiemap": "false", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fiemap_threshold": "4096", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-26T01:23:56.700 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_inject_stall": "0", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_journal_parallel": "false", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_journal_trailing": "false", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_journal_writeahead": "false", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_kill_at": "0", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs": "0", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-26T01:23:56.701 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size": "0", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_merge_threshold": "-10", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_odsync_write": "false", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_omap_backend": "rocksdb", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_omap_backend_path": "", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_op_thread_timeout": "60", 2024-04-26T01:23:56.702 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_op_threads": "2", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_punch_hole": "false", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_queue_max_ops": "50", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_seek_data_hole": "false", 2024-04-26T01:23:56.703 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_sloppy_crc": "false", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_splice": "false", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_split_multiple": "2", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_split_rand_factor": "20", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_update_to": "1000", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_enable": "true", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-26T01:23:56.704 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_xfs_extsize": "false", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "filestore_zfs_snap": "false", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fio_dir": "/tmp/fio", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fsid": "ec0903d9-9475-4d3a-af50-1dbb367f53fb", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_allow_other": "true", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_atomic_o_trunc": "true", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_big_writes": "true", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_debug": "false", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_default_permissions": "false", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_disable_pagecache": "false", 2024-04-26T01:23:56.705 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_max_write": "0", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_multithreaded": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_require_active_mds": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_set_user_groups": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_splice_move": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_splice_read": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_splice_write": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "fuse_use_invalidate_cb": "true", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "gss_target_name": "ceph", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "heartbeat_file": "", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "heartbeat_inject_failure": "0", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "heartbeat_interval": "5", 2024-04-26T01:23:56.706 INFO:teuthology.orchestra.run.smithi037.stdout: "host": "smithi037", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "inject_early_sigterm": "false", 2024-04-26T01:23:56.707 INFO:teuthology.orchestra.run.smithi037.stdout: "jaeger_agent_port": "6799", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "jaeger_tracing_enable": "false", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_aio": "true", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_align_min_size": "65536", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_block_align": "true", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_block_size": "4096", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_dio": "true", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_discard": "false", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_force_aio": "false", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_ignore_corruption": "false", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_max_write_bytes": "10485760", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_max_write_entries": "100", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_replay_from": "0", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-26T01:23:56.708 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_write_header_frequency": "0", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "journal_zero_on_create": "false", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "journaler_prefetch_periods": "10", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "journaler_prezero_periods": "5", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "journaler_write_head_interval": "15", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "key": "", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "keyfile": "", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_backend": "rocksdb", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_default_stripe_size": "65536", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_fsck_on_mount": "false", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_max_bytes": "67108864", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_max_ops": "512", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_nid_prealloc": "1024", 2024-04-26T01:23:56.709 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_onode_map_size": "1024", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_sync_submit_transaction": "false", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "kstore_sync_transaction": "false", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "librados_thread_count": "2", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "lockdep": "false", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "lockdep_force_backtrace": "false", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_coarse_timestamps": "true", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_flush_on_exit": "false", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_graylog_host": "127.0.0.1", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_graylog_port": "12201", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_max_new": "1000", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_max_recent": "10000", 2024-04-26T01:23:56.710 INFO:teuthology.orchestra.run.smithi037.stdout: "log_stderr_prefix": "", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "log_stop_at_utilization": "0.970000", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_file": "true", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_graylog": "false", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_journald": "false", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_stderr": "false", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "log_to_syslog": "false", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "max_rotating_auth_attempts": "10", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_action_on_write_error": "1", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_alternate_name_max": "8192", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_asio_thread_count": "2", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_export_pin": "true", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_dirs": "true", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_interval": "5", 2024-04-26T01:23:56.711 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_interval": "10", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_max": "-1", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_max_until": "-1", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_merge_size": "50", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_midchunk": "0.300000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_min_start": "0.200000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_minchunk": "0.001000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_mode": "0", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_need_max": "1.200000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_need_min": "0.800000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_overload_epochs": "2", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-26T01:23:56.712 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_bits": "3", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_size": "10000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_target_decay": "10.000000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_beacon_grace": "15.000000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_beacon_interval": "4.000000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_mid": "0.700000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_delay": "0", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-26T01:23:56.713 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_release_free_interval": "10", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_reservation": "0.050000", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_trim_interval": "1", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cache_trim_threshold": "262144", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_client_prealloc_inos": "1000", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_connect_bootstrapping": "false", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_damage_table_max_entries": "10000", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_auth_pins": "false", 2024-04-26T01:23:56.714 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_frag": "false", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_scatterstat": "false", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_debug_subtrees": "false", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_decay_halflife": "5.000000", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_default_dir_hash": "2", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_defer_session_stale": "true", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_deny_all_reconnect": "false", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_keys_per_op": "16384", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_max_commit_size": "10", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_max_entries": "0", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dir_prefetch": "true", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_on_map": "false", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-26T01:23:56.715 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_early_reply": "true", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_enable_op_tracker": "true", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_enforce_unique_name": "true", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_random": "true", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_health_summarize_threshold": "10", 2024-04-26T01:23:56.716 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_health_dummy": "false", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_migrator_session_race": "false", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_join_fs": "", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_journal_format": "1", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_create_at": "0", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_export_at": "0", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_import_at": "0", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_journal_at": "0", 2024-04-26T01:23:56.717 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_journal_expire_at": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_journal_replay_at": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_link_at": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_mdstable_at": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_openc_at": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_rename_at": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_shutdown_at": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_event_large_threshold": "524288", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_events_per_segment": "1024", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_max_events": "-1", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_max_segments": "128", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_pause": "false", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_segment_size": "0", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-26T01:23:56.718 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_trim_threshold": "128", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_log_warn_factor": "2.000000", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_caps_per_client": "1048576", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_completed_flushes": "100000", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_completed_requests": "100000", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_export_size": "20971520", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_file_recover": "32", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_purge_files": "64", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_purge_ops": "8192", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-26T01:23:56.719 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_max_snaps_per_dir": "100", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_metrics_update_interval": "2", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_min_caps_per_client": "100", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_min_caps_working_set": "10000", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_numa_node": "-1", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_complaint_time": "30.000000", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_duration": "600", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_size": "20", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_slow_op_size": "20", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_op_log_threshold": "5", 2024-04-26T01:23:56.720 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_ping_grace": "15", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_ping_interval": "5", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_max_caps": "30000", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_recall_warning_threshold": "262144", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_replay_interval": "1.000000", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_root_ino_gid": "0", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_root_ino_uid": "0", 2024-04-26T01:23:56.721 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_shutdown_check": "0", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_skip_ino": "0", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-26T01:23:56.722 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_snap_max_uid": "4294967294", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_snap_min_uid": "0", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_snap_rstat": "false", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_standby_replay_damaged": "false", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_symlink_recovery": "true", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_thrash_exports": "0", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_thrash_fragments": "0", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_tick_interval": "5.000000", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_valgrind_exit": "false", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_verify_backtrace": "1", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_verify_scatter": "false", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mds_wipe_sessions": "false", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "mempool_debug": "false", 2024-04-26T01:23:56.723 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_device_bytes": "1073741824", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_page_set": "false", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "memstore_page_size": "65536", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_client_bytes": "134217728", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_client_messages": "512", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_disabled_modules": "", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-26T01:23:56.724 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_max_pg_num_change": "128", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mds_bytes": "134217728", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mds_messages": "128", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mon_bytes": "134217728", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_mon_messages": "128", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_osd_bytes": "536870912", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_osd_messages": "8192", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_pool": "true", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_standby_modules": "true", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_stats_period": "5", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_stats_threshold": "5", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_tick_period": "2", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-26T01:23:56.725 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_allow_pool_delete": "true", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_allow_pool_size_one": "true", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_auth_validate_all_caps": "true", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_bytes": "104857600", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_directed_command_retry": "2", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-26T01:23:56.726 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_hunt_parallel": "3", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_log_interval": "1.000000", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_ping_interval": "10.000000", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_client_target_rank": "-1", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_file_level": "debug", 2024-04-26T01:23:56.727 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_file": "true", 2024-04-26T01:23:56.729 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_journald": "false", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_compact_on_bootstrap": "false", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_compact_on_start": "false", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_compact_on_trim": "true", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_cpu_threads": "4", 2024-04-26T01:23:56.730 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_crush_min_required_version": "hammer", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_daemon_bytes": "419430400", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data_avail_crit": "5", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data_avail_warn": "5", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_data_size_warn": "16106127360", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_dump_json": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_dump_transactions": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_extra_checks": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_require_reef": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_no_require_squid": "false", 2024-04-26T01:23:56.731 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_down_mkfs_grace": "120", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_down_uptime_grace": "60", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_election_default_strategy": "3", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_election_timeout": "5.000000", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_elector_ping_divisor": "2", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_enable_op_tracker": "true", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_fake_pool_delete": "false", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_force_quorum_join": "false", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_globalid_prealloc": "10000", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_detail_to_clog": "false", 2024-04-26T01:23:56.732 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_log_update_period": "5", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_max_detail": "50", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_to_clog": "true", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_to_clog_interval": "600", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_host": "172.21.15.37,172.21.15.45,172.21.15.71", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_host_override": "", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_initial_members": "", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_lease": "5.000000", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-26T01:23:56.733 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_log_full_interval": "50", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_log_max": "10000", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_log_max_summary": "50", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_log_epochs": "500", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_osd": "10000", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_pg_per_osd": "10000", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_pool_pg_num": "65536", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-26T01:23:56.734 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mds_force_trim_to": "0", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mds_skip_sanity": "false", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_memory_autotune": "true", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_memory_target": "2147483648", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_beacon_grace": "30", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_digest_period": "5", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_inactive_grace": "60", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_min_osdmap_epochs": "500", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_complaint_time": "30", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_duration": "600", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_size": "20", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_slow_op_size": "20", 2024-04-26T01:23:56.735 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_op_log_threshold": "5", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_auto_mark_in": "false", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_cache_size": "500", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-26T01:23:56.736 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_down_out_interval": "600", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_force_trim_to": "0", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_max_creating_pgs": "1024", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_min_down_reporters": "2", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-26T01:23:56.737 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_report_timeout": "900", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-26T01:23:56.738 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_stuck_threshold": "60", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_probe_timeout": "2.000000", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_max_change": "0.050000", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_max_osds": "4", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-26T01:23:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_interval": "300", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_max_keys": "100", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_scrub_timeout": "300", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_session_timeout": "300", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_smart_report_timeout": "5", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stat_smooth_intervals": "6", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_pool_min_size": "2", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_pool_size": "4", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_debug": "false", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_provider_kill_at": "0", 2024-04-26T01:23:56.740 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_requester_kill_at": "0", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_sync_timeout": "60.000000", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_target_pg_per_osd": "100", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_tick_interval": "5", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_timecheck_interval": "300.000000", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_use_min_delay_socket": "false", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_older_version_delay": "604800", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-26T01:23:56.741 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_misplaced": "false", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_older_version": "true", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_pool_no_app": "true", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-26T01:23:56.742 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "monmap": "", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "motr_tracing_enabled": "false", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_op_threads": "3", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_cm": "false", 2024-04-26T01:23:56.743 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_device_name": "", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_dscp": "96", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_local_gid": "", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_port_num": "1", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_sl": "3", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_support_srq": "true", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_rdma_type": "ib", 2024-04-26T01:23:56.744 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_async_reap_threshold": "5", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_before_connect": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_ipv4": "true", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_ipv6": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_msgr1": "true", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_msgr2": "true", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_port_max": "7568", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_port_min": "6800", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_retry_count": "3", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_bind_retry_delay": "5", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_client": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_mds": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_mgr": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_mon": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_blackhole_osd": "false", 2024-04-26T01:23:56.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_client_mode": "crc secure", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_cluster_mode": "crc secure", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_cluster_type": "", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_compress_secure": "false", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_connection_idle_timeout": "900", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_connection_ready_timeout": "10", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_crc_data": "true", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_crc_header": "true", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_bad_msg": "false", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_bug": "true", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_old_message": "true", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_skipped_message": "false", 2024-04-26T01:23:56.746 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_coremask": "0xF", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_enable_tso": "true", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_hugepages": "", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_lro": "true", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_memory_channel": "4", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-26T01:23:56.747 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_pmd": "", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_port_id": "0", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_dump_on_send": "false", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_initial_backoff": "0.200000", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_delay_max": "1.000000", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_delay_type": "", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_network_congestion": "0", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_inject_socket_failures": "0", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_learn_addr_from_peer": "true", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_max_accept_failures": "4", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_max_backoff": "15.000000", 2024-04-26T01:23:56.748 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_mon_client_mode": "secure crc", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_mon_service_mode": "secure crc", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_osd_compress_min_size": "1024", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_osd_compress_mode": "none", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_pq_min_cost": "65536", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_public_type": "", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_service_mode": "crc secure", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_listen_backlog": "512", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_nodelay": "true", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_tcp_rcvbuf": "0", 2024-04-26T01:23:56.749 INFO:teuthology.orchestra.run.smithi037.stdout: "ms_type": "async+posix", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "no_config_file": "false", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_completion_locks_per_session": "32", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_inflight_ops": "1024", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_tick_interval": "5.000000", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objecter_timeout": "10.000000", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objectstore_blackhole": "false", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "openssl_engine_opts": "", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_delay_time": "5.000000", 2024-04-26T01:23:56.750 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_hist_halflife": "1000", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_max_low_ops": "2", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_max_ops": "4", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_agent_slop": "0.020000", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_asio_thread_count": "2", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_async_recovery_min_cost": "100", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backfill_scan_max": "512", 2024-04-26T01:23:56.751 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backfill_scan_min": "64", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backoff_on_degraded": "false", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backoff_on_peering": "false", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_backoff_on_unfound": "true", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_beacon_report_interval": "300", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_duration": "30", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_max_block_size": "67108864", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_blkin_trace_all": "false", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-26T01:23:56.752 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_check_for_log_corruption": "false", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_class_update_on_start": "true", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_message_cap": "256", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_message_size_cap": "524288000", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_op_priority": "63", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_client_watch_timeout": "30", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_command_max_records": "256", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_compact_on_start": "false", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-26T01:23:56.753 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_update_on_start": "true", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_crush_update_weight_set": "true", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_feed_pullee": "-1", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_misdirected_ops": "false", 2024-04-26T01:23:56.754 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_no_acting_change": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_no_purge_strays": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_op_order": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_shutdown": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_trim_objects": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-26T01:23:56.755 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_snaps": "false", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_keys": "1024", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_stride": "524288", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_default_notify_timeout": "30", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep": "0.000000", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-26T01:23:56.756 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_discard_disconnected_ops": "true", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_enable_op_tracker": "true", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_info": "true", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_shutdown": "true", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_function_tracing": "false", 2024-04-26T01:23:56.757 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_grace": "20", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_interval": "6", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_min_peers": "10", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_min_size": "2000", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_stale": "600", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_hit_set_max_size": "100000", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_hit_set_min_size": "1000", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-26T01:23:56.758 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_journal_size": "5120", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_kill_backfill_at": "0", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_cache_size": "50", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_dedup": "true", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_message_max": "40", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_map_share_max_epochs": "40", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_attr_name_len": "100", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_attr_size": "0", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_backfills": "1", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_markdown_count": "5", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_markdown_period": "600", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_object_name_len": "2048", 2024-04-26T01:23:56.759 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_object_namespace_len": "256", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_object_size": "134217728", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pg_blocked_by": "16", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pg_log_entries": "10000", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_pgls": "1024", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_push_cost": "8388608", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_push_objects": "10", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_scrubs": "3", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_trimming_pgs": "2", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_write_op_reply_len": "64", 2024-04-26T01:23:56.760 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_max_write_size": "90", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_profile": "balanced", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-26T01:23:56.761 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_base": "805306368", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_cache_min": "134217728", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_target": "4294967296", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_target_autotune": "false", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_min_pg_log_entries": "250", 2024-04-26T01:23:56.762 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_min_recovery_priority": "0", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_report_interval": "5", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_num_cache_shards": "32", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_num_op_tracker_shard": "32", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_numa_auto_affinity": "true", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_numa_node": "-1", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_numa_prefer_iface": "true", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objecter_finishers": "1", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objectstore": "bluestore", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objectstore_fuse": "false", 2024-04-26T01:23:56.763 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_objectstore_tracing": "false", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_complaint_time": "30.000000", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_duration": "600", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_size": "20", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_slow_op_size": "20", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_log_threshold": "5", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_shards": "0", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_shards_hdd": "5", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_shards_ssd": "8", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-26T01:23:56.764 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_pq_min_cost": "65536", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_queue_cut_off": "high", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_op_thread_timeout": "15", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_open_classes_on_start": "true", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_os_flags": "0", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_peering_op_priority": "255", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_delete_cost": "1048576", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_delete_priority": "5", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_log_trim_max": "10000", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_log_trim_min": "100", 2024-04-26T01:23:56.765 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_crimson": "false", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-26T01:23:56.766 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_flags": "0", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_min_size": "0", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_pg_num": "32", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_pgp_num": "0", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_read_ratio": "70", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_size": "2", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_default_type": "replicated", 2024-04-26T01:23:56.767 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_push_per_object_cost": "1000", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recover_clone_overlap": "true", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_cost": "20971520", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_active": "0", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_max_single_start": "1", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_op_priority": "3", 2024-04-26T01:23:56.768 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_priority": "5", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep": "0.000000", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_repair_during_recovery": "false", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_requested_scrub_priority": "120", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_auto_repair": "false", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-26T01:23:56.769 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_begin_hour": "0", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_begin_week_day": "0", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_chunk_max": "25", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_chunk_min": "5", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_cost": "52428800", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_disable_reservation_queuing": "true", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_during_recovery": "false", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_end_hour": "0", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_end_week_day": "0", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_event_cost": "4096", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_invalid_stats": "true", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-26T01:23:56.770 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_max_preemptions": "5", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_priority": "5", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_scrub_sleep": "0.000000", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_skip_data_digest": "false", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_smart_report_timeout": "5", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_cost": "1048576", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_priority": "5", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-26T01:23:56.771 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_target_transaction_size": "30", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-26T01:23:56.772 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_tracing": "false", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_use_stale_snap": "false", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "osdc_blkin_trace_all": "false", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_kill_at": "0", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_max_join_drift": "10", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_min": "500", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_min_wait": "0.050000", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_propose_interval": "1.000000", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_service_trim_max": "500", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_service_trim_min": "250", 2024-04-26T01:23:56.773 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_stash_full_interval": "25", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_trim_max": "500", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "paxos_trim_min": "250", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "perf": "true", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "plugin_dir": "/usr/lib/ceph", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addr": "-", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "public_addrv": "", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "public_bind_addr": "-", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "public_network": "", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "public_network_interface": "", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "qat_compressor_enabled": "false", 2024-04-26T01:23:56.774 INFO:teuthology.orchestra.run.smithi037.stdout: "qat_compressor_session_max_number": "256", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rados_mon_op_timeout": "0", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rados_osd_op_timeout": "0", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rados_tracing": "false", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_atime_update_interval": "60", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_balance_parent_reads": "false", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_balance_snap_reads": "false", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_blkin_trace_all": "false", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache": "true", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-26T01:23:56.775 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_policy": "writearound", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_size": "33554432", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_clone_copy_on_read": "false", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_compression_hint": "none", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_concurrent_management_ops": "10", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_clone_format": "auto", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_data_pool": "", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_features": "61", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_format": "2", 2024-04-26T01:23:56.776 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_map_options": "", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_order": "22", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_pool": "rbd", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_stripe_count": "0", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_default_stripe_unit": "0", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_enable_alloc_hint": "true", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_io_scheduler": "simple", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-26T01:23:56.777 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_order": "24", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_pool": "", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_journal_splay_width": "4", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_localize_parent_reads": "false", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_localize_snap_reads": "false", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-26T01:23:56.778 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-26T01:23:56.779 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_mtime_update_interval": "60", 2024-04-26T01:23:56.780 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_non_blocking_aio": "true", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_op_thread_timeout": "60", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_op_threads": "1", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_parent_cache_enabled": "false", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_plugins": "", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_bps_burst": "0", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-26T01:23:56.781 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_bps_limit": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_exclude_ops": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_iops_burst": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_iops_limit": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-26T01:23:56.782 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_read_from_replica_policy": "default", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_skip_partial_discard": "true", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-26T01:23:56.783 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_tracing": "false", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_validate_names": "true", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rbd_validate_pool": "true", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "restapi_base_url": "", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "restapi_log_level": "", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_acl_grants_max_num": "100", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_admin_entry": "admin", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_backend_store": "rados", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_barbican_url": "", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_beast_enable_async": "true", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_counters_cache": "false", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-26T01:23:56.784 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cache_enabled": "true", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cache_expiry_interval": "900", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cache_lru_size": "10000", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_config_store": "rados", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_content_length_compat": "false", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_copy_obj_progress": "true", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cors_rules_max_num": "100", 2024-04-26T01:23:56.785 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_cross_domain_policy": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_addr": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_password": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_kmip_username": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_require_ssl": "true", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-26T01:23:56.786 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_addr": "", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_auth": "token", 2024-04-26T01:23:56.787 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_namespace": "", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_prefix": "", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_token_file": "", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_buffersize": "524288", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_low_speed_time": "300", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-26T01:23:56.788 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_d4n_port": "6379", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_changes_size": "1000", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_num_shards": "128", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_log_window": "30", 2024-04-26T01:23:56.789 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-26T01:23:56.790 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dns_name": "", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dns_s3website_name": "", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_dynamic_resharding": "true", 2024-04-26T01:23:56.791 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_gc_threads": "true", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_lc_threads": "true", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_ops_log": "false", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_quota_threads": "true", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_static_website": "false", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enable_usage_log": "false", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_enforce_swift_acls": "true", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_exit_timeout_secs": "120", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_expose_bucket": "false", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_extended_http_attrs": "", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_filter": "none", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_frontends": "beast port=7480", 2024-04-26T01:23:56.792 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_deferred": "50", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_objs": "32", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_gc_processor_period": "3600", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_init_timeout": "300", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-26T01:23:56.793 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_kafka_connection_idle": "300", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_domain": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_password": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_password_path": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_project": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_tenant": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_token": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_token_path": "", 2024-04-26T01:23:56.794 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_admin_user": "", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_api_version": "2", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_domain": "", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_password": "", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_project": "", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_barbican_user": "", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_url": "", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_debug_interval": "-1", 2024-04-26T01:23:56.795 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_lock_max_time": "90", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_objs": "32", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_rules": "1000", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_worker": "3", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lc_thread_delay": "0", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_dnattr": "uid", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_searchfilter": "", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ldap_uri": "", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-26T01:23:56.796 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_http_headers": "", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_log_object_name_utc": "false", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_attr_name_len": "0", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_attr_size": "0", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_chunk_size": "4194304", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-26T01:23:56.797 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_listing_results": "1000", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_notify_retries": "10", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_put_param_size": "1048576", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_put_size": "5368709120", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_max_slo_entries": "1000", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_md_log_max_shards": "64", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-26T01:23:56.798 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_mp_lock_max_time": "600", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_max_gc": "300", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-26T01:23:56.799 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_num_async_rados_threads": "32", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_num_control_oids": "8", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_numa_node": "-1", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_objexp_chunk_size": "100", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_objexp_gc_interval": "600", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-26T01:23:56.800 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_op_thread_timeout": "600", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_op_tracing": "false", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_opa_token": "", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_opa_url": "", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_opa_verify_ssl": "true", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_rados": "false", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_ops_log_socket_path": "", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-26T01:23:56.801 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_push_interval": "2.000000", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_lanes": "3", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_cache_partitions": "3", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_print_continue": "true", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-26T01:23:56.802 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rados_tracing": "false", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_realm": "", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_realm_id": "", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_region": "", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_request_uri": "", 2024-04-26T01:23:56.803 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_batch_size": "64", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_max_aio": "128", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_num_logs": "16", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_reshard_thread_interval": "600", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_resolve_cname": "false", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_run_sync_thread": "true", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-26T01:23:56.804 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_scheduler_type": "throttler", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_script_uri": "", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_service_provider_name": "", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_client_id": "", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_client_secret": "", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_entry": "sts", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_key": "sts", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_min_session_duration": "900", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sts_token_introspection_url": "", 2024-04-26T01:23:56.805 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_account_in_url": "false", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_auth_entry": "auth", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_auth_url": "", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_custom_header": "", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_need_stats": "true", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_tenant_name": "", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_token_expiration": "86400", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_url": "", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_url_prefix": "swift", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_lease_period": "120", 2024-04-26T01:23:56.806 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_thread_pool_size": "512", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-26T01:23:56.807 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_comment": "", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_createby": "", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_encoding": "", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_flag": "false", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_origin": "", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_torrent_tracker": "", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_trust_forwarded_https": "false", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_max_shards": "32", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_usage_max_user_shards": "1", 2024-04-26T01:23:56.808 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_use_opa_authz": "false", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_counters_cache": "false", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_max_buckets": "1000", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_policies_max_num": "100", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_user_unique_email": "true", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_verify_ssl": "true", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-26T01:23:56.809 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zone": "", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zone_id": "", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zonegroup": "", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zonegroup_id": "", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rgwlc_auto_session_clear": "true", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_block_size": "4096", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-26T01:23:56.810 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_size": "536870912", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_index_type": "binary_search", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_partition_filters": "false", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_perf": "false", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-26T01:23:56.811 INFO:teuthology.orchestra.run.smithi037.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "run_dir": "/var/run/ceph", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_block_create": "true", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_cache_lru_size": "67108864", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_default_max_object_size": "16777216", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_device_size": "53687091200", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_batch_capacity": "16", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_main_device_type": "SSD", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-26T01:23:56.812 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "seastore_segment_size": "67108864", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "set_keepcaps": "false", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "setgroup": "", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "setuser": "", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "setuser_match_path": "", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "thp": "false", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "threadpool_default_timeout": "60", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-26T01:23:56.813 INFO:teuthology.orchestra.run.smithi037.stdout: "throttler_perf_counter": "true" 2024-04-26T01:23:56.814 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:23:56.816 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-26T01:23:56.963 INFO:teuthology.orchestra.run.smithi037.stdout:mon.a: {} 2024-04-26T01:23:56.963 INFO:teuthology.orchestra.run.smithi037.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:23:56.976 INFO:teuthology.orchestra.run.smithi037.stdout:mon.b: {} 2024-04-26T01:23:56.976 INFO:teuthology.orchestra.run.smithi037.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:23:56.988 INFO:teuthology.orchestra.run.smithi037.stdout:mon.c: {} 2024-04-26T01:23:56.988 INFO:teuthology.orchestra.run.smithi037.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:23:57.004 DEBUG:teuthology.orchestra.run.smithi037:> type ceph-objectstore-tool 2024-04-26T01:23:57.007 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:23:57.008 DEBUG:teuthology.orchestra.run.smithi181:> type ceph-objectstore-tool 2024-04-26T01:23:57.013 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:23:57.013 DEBUG:teuthology.orchestra.run.smithi045:> type ceph-objectstore-tool 2024-04-26T01:23:57.019 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:23:57.019 DEBUG:teuthology.orchestra.run.smithi071:> type ceph-objectstore-tool 2024-04-26T01:23:57.025 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-26T01:23:57.025 INFO:teuthology.run_tasks:Running task rados... 2024-04-26T01:23:57.036 INFO:tasks.rados:Beginning rados... 2024-04-26T01:23:57.036 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-04-26T01:23:57.098 INFO:tasks.rados:joining rados 2024-04-26T01:23:57.098 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-26T01:23:57.098 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:23:57.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:23:57.099 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-04-26T01:23:57.099 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-04-26T01:23:57.099 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-26T01:23:57.100 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-26T01:23:57.100 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:23:57.100 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-26T01:23:57.100 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:23:57.101 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-26T01:23:57.101 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:23:57.101 INFO:tasks.rados:clients are ['client.0'] 2024-04-26T01:23:57.101 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-04-26T01:23:57.103 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:23:57.100+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:23:57.201 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:23:57.200+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:23:57.248 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:23:57.249 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:23:57.269 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:23:57.269 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:23:57.281 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:57.281 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:57.295 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:23:57.302 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:23:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:23:57.303 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:23:57.304+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:23:57.325 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:23:57.325 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:23:57.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:23:57.342 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:23:57.362 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:23:57.363 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:23:57.382 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:23:57.383 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:23:57.403 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:23:57.408+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:23:57.418 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:23:57.418 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:23:57.418 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T01:23:57.420+0000 7f533fe62640 -1 mon.a@0(leader).osd e27 definitely_dead 0 2024-04-26T01:23:57.434 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:57.434 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:57.447 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:23:57.450 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:23:57.451 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:23:57.485 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:23:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:23:57.504 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:23:57.508+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:23:57.511 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:23:57.511 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:23:57.532 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:23:57.532 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:23:57.549 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:23:57.549 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:23:57.569 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:57.569 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:57.570 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:23:57.570 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:23:57.582 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:23:57.591 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:23:57.591 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:23:57.604 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:23:57.604+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:23:57.611 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:23:57.611 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:23:57.704 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:57.705 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:57.705 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:23:57.708+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:23:57.717 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:23:57.806 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:57.806+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:23:57.842 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:57.842 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:57.855 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:23:57.907 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:23:57.912+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:23:57.949 INFO:teuthology.orchestra.run.smithi037.stderr:load: shec 2024-04-26T01:23:57.949 INFO:teuthology.orchestra.run.smithi037.stderr:marked out osd.7. 2024-04-26T01:23:57.950 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-04-26T01:23:57.977 INFO:tasks.rados:starting run 0 out of 1 2024-04-26T01:23:57.977 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-04-26T01:23:57.977 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-04-26T01:23:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:57.991 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:58.005 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:23:58.008 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:23:58.012+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:23:58.109 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:23:58.112+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:23:58.162 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:58.162 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:58.176 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:23:58.219 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:23:58.224+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:23:58.299 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:58.299 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:58.312 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:23:58.312 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:23:58.312+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:23:58.412 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:23:58.416+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:23:58.478 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:58.479 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:58.492 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:23:58.514 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:23:58.516+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:23:58.615 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:58.614+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:23:58.657 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:58.657 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:58.671 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:23:58.716 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:58.718+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:23:58.798 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:58.798 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:58.812 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:23:58.816 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:23:58.822+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:23:58.917 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:23:58.916+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:23:58.939 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:58.939 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:58.954 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:23:59.017 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:23:59.018+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:23:59.089 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:59.090 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:59.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:23:59.119 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:23:59.120+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:23:59.219 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:23:59.224+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:23:59.229 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:59.229 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:59.244 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:23:59.320 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:23:59.320+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:23:59.371 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:59.372 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:59.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:23:59.422 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:23:59.424+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:23:59.523 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:23:59.524+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:23:59.561 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:59.561 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:59.575 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:23:59.624 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:23:59.626+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:23:59.628 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:23:59.725 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:23:59.726+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:23:59.746 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:23:59.746 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:23:59.760 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:23:59.788 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:23:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:23:59.807 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:23:59.807 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:23:59.826 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:23:59.828+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:23:59.828 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:23:59.829 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:23:59.851 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:23:59.851 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:23:59.868 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:23:59.868 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:23:59.889 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:23:59.889 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:23:59.890 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:23:59.890 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:23:59.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:23:59.890 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [] 2024-04-26T01:23:59.890 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:23:59.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:23:59.911 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:23:59.911 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:23:59.927 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:23:59.926+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:23:59.933 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:23:59.933 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:23:59.953 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:23:59.954 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:23:59.956 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'unique_pool_0' created 2024-04-26T01:23:59.980 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:23:59.980 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:23:59.985 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:23:59.988 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-04-26T01:24:00.005 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:00.005 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:00.027 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:00.027 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:00.028 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:00.032+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:00.032 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:00.032 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:00.032 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-04-26T01:24:00.032 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:00.050 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:00.107 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:00.108 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:00.128 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:00.132+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:00.136 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:00.137 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:00.159 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:00.175 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:00.176 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:00.176 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:00.176 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:00.176 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:00.177 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:00.177 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:00.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:00.238 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:00.240+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:00.318 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:00.318 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:00.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:00.319 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [] 2024-04-26T01:24:00.319 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:00.330 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:00.330+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:00.333 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:00.432 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:00.434+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:172 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [call lock.lock in=106b,cmpxattr striper.excl op 1 mode 1 in=12b,setxattr striper.excl (25) in=37b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:00.380787+0000", 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.124660808, 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.124718754, 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.500 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for sub ops", 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 172 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.380787+0000", 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.380787+0000", 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.380792+0000", 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3630000000000004e-06 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.380817+0000", 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5579e-05 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.380864+0000", 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6310999999999998e-05 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.380880+0000", 2024-04-26T01:24:00.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6294e-05 2024-04-26T01:24:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.481172+0000", 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10029221300000001 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.481459+0000", 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028684099999999997 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.482087+0000", 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00062800599999999996 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.483490+0000", 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014033629999999999 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 1 2024-04-26T01:24:00.506 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:00.514 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:00.532 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:00.536+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:00.634 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:00.636+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:00.635 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:00.635 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:00.635 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:00.635 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:00.636 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:00.651 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:00.735 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:00.740+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:00.818 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:00.818 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:00.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.1888701729999998, 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40356932699999998, 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.819 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 37 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4700000000000001e-07 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630292+0000", 2024-04-26T01:24:00.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3669999999999999e-06 2024-04-26T01:24:00.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630302+0000", 2024-04-26T01:24:00.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0115999999999999e-05 2024-04-26T01:24:00.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630310+0000", 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2029999999999992e-06 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735212+0000", 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104902254 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735342+0000", 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012980500000000001 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735542+0000", 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020034 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.736506+0000", 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00096400499999999998 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033692+0000", 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29718516499999997 2024-04-26T01:24:00.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033783+0000", 2024-04-26T01:24:00.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1377000000000003e-05 2024-04-26T01:24:00.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033853+0000", 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9948000000000002e-05 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:72 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.308421+0000", 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.5107329099999998, 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30681187900000001, 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.827 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 72 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308421+0000", 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308421+0000", 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308422+0000", 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2899999999999999e-06 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308574+0000", 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015142500000000001 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308592+0000", 2024-04-26T01:24:00.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8468999999999999e-05 2024-04-26T01:24:00.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.830 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.830 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308603+0000", 2024-04-26T01:24:00.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0876000000000001e-05 2024-04-26T01:24:00.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.509941+0000", 2024-04-26T01:24:00.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20133832500000001 2024-04-26T01:24:00.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.510077+0000", 2024-04-26T01:24:00.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013609599999999999 2024-04-26T01:24:00.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.510289+0000", 2024-04-26T01:24:00.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000212093 2024-04-26T01:24:00.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.513246+0000", 2024-04-26T01:24:00.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029563990000000002 2024-04-26T01:24:00.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.615017+0000", 2024-04-26T01:24:00.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101771323 2024-04-26T01:24:00.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.615106+0000", 2024-04-26T01:24:00.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9073000000000006e-05 2024-04-26T01:24:00.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.615233+0000", 2024-04-26T01:24:00.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012651000000000001 2024-04-26T01:24:00.837 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.837 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.837 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.838 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:73 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.838 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.308629+0000", 2024-04-26T01:24:00.838 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.510524771, 2024-04-26T01:24:00.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.306862725, 2024-04-26T01:24:00.838 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.838 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.839 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.839 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.839 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.839 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.839 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 73 2024-04-26T01:24:00.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.840 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308629+0000", 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308629+0000", 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308630+0000", 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.017e-06 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308773+0000", 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014341500000000001 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308790+0000", 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6857000000000001e-05 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308800+0000", 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7790000000000007e-06 2024-04-26T01:24:00.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.511079+0000", 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20227899099999999 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.511225+0000", 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014618500000000001 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.511426+0000", 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020052700000000001 2024-04-26T01:24:00.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.513265+0000", 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0018394259999999999 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.615332+0000", 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102066403 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.615423+0000", 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1553999999999998e-05 2024-04-26T01:24:00.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.615492+0000", 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8571000000000005e-05 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.308818+0000", 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.5103358330000001, 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30741775900000001, 2024-04-26T01:24:00.845 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 74 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308818+0000", 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308818+0000", 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308819+0000", 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.003e-06 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308827+0000", 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3760000000000008e-06 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308840+0000", 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3006e-05 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.308849+0000", 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2940000000000006e-06 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.512224+0000", 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203375431 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.512350+0000", 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000126069 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.512585+0000", 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023498900000000001 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.513467+0000", 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00088141700000000001 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.616080+0000", 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102613465 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.616171+0000", 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0686999999999994e-05 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.616236+0000", 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5022000000000002e-05 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.850 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.8877638880000003, 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40787895499999999, 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 84 2024-04-26T01:24:00.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931392+0000", 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7659999999999999e-06 2024-04-26T01:24:00.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931546+0000", 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015429199999999999 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931571+0000", 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5381999999999999e-05 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931597+0000", 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5616e-05 2024-04-26T01:24:00.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934003+0000", 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024058180000000001 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934128+0000", 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505200000000001 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934351+0000", 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000222912 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935791+0000", 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014398799999999999 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339098+0000", 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40330716900000002 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339195+0000", 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7676999999999993e-05 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339269+0000", 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3391000000000003e-05 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:00.856 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.8875216830000001, 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.407895061, 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 85 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931634+0000", 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4330000000000001e-06 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931646+0000", 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2622999999999999e-05 2024-04-26T01:24:00.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931665+0000", 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8669999999999999e-05 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931699+0000", 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4112000000000001e-05 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934956+0000", 2024-04-26T01:24:00.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032571119999999999 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935079+0000", 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122483 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935269+0000", 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019042700000000001 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.936573+0000", 2024-04-26T01:24:00.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013043340000000001 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339389+0000", 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.402815335 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339468+0000", 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9758000000000004e-05 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339527+0000", 2024-04-26T01:24:00.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8773999999999997e-05 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.8874266889999998, 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40806487499999999, 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.862 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 86 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931728+0000", 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3030000000000001e-06 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931739+0000", 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.046e-05 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931758+0000", 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8842e-05 2024-04-26T01:24:00.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.031855+0000", 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100097588 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032039+0000", 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000184141 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032212+0000", 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000172747 2024-04-26T01:24:00.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032428+0000", 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021604 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.135784+0000", 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10335620700000001 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339636+0000", 2024-04-26T01:24:00.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20385123199999999 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339735+0000", 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9036999999999994e-05 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339792+0000", 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7278000000000003e-05 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:87 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:55.031912+0000", 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.7872415039999998, 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30828704699999998, 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 87 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.031912+0000", 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.031912+0000", 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.031914+0000", 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.643e-06 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032017+0000", 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103256 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032039+0000", 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1359999999999999e-05 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032055+0000", 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6262000000000001e-05 2024-04-26T01:24:00.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.133057+0000", 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10100181499999999 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.133214+0000", 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015713200000000001 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.133419+0000", 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020510299999999999 2024-04-26T01:24:00.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.135802+0000", 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023833119999999998 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.340074+0000", 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.204271482 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.340147+0000", 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3559999999999994e-05 2024-04-26T01:24:00.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.340199+0000", 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2122e-05 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.7870547569999999, 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40781067500000001, 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032101+0000", 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7630000000000001e-06 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032115+0000", 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4375999999999999e-05 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032137+0000", 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1356e-05 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032158+0000", 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1253000000000001e-05 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134605+0000", 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102446813 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134733+0000", 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012879999999999999 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134935+0000", 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020153599999999999 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.136395+0000", 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014603089999999999 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439763+0000", 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30336767999999997 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439846+0000", 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3430999999999995e-05 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439910+0000", 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3357999999999999e-05 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405207+0000", 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4139469719999997, 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.308960927, 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 107 2024-04-26T01:24:00.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405207+0000", 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405207+0000", 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405208+0000", 2024-04-26T01:24:00.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8299999999999995e-07 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405214+0000", 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6000000000000003e-06 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405225+0000", 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1005999999999999e-05 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405233+0000", 2024-04-26T01:24:00.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1359999999999997e-06 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.409297+0000", 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040637110000000002 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.409532+0000", 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000234742 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.409912+0000", 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000380178 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414844+0000", 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049318599999999997 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.713884+0000", 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29904050700000001 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714051+0000", 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166754 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714168+0000", 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011655 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405271+0000", 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4138826409999998, 2024-04-26T01:24:00.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30920326599999998, 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 108 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405271+0000", 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405271+0000", 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405272+0000", 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405279+0000", 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6599999999999998e-06 2024-04-26T01:24:00.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405291+0000", 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2211999999999999e-05 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405300+0000", 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7669999999999992e-06 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.411128+0000", 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058281779999999998 2024-04-26T01:24:00.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.411372+0000", 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024450900000000001 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.411759+0000", 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038703700000000001 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417160+0000", 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.005400658 2024-04-26T01:24:00.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714317+0000", 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29715656299999998 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714407+0000", 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0061000000000004e-05 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714474+0000", 2024-04-26T01:24:00.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7654e-05 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405315+0000", 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4138382820000004, 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30982507199999998, 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 109 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405315+0000", 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405315+0000", 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405316+0000", 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405436+0000", 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011977999999999999 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405450+0000", 2024-04-26T01:24:00.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3706000000000001e-05 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405487+0000", 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6869000000000002e-05 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.412356+0000", 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068690410000000002 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.412602+0000", 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024654199999999999 2024-04-26T01:24:00.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.413004+0000", 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00040204100000000001 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417187+0000", 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041823590000000001 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.715000+0000", 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.2978131 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.715080+0000", 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9993999999999993e-05 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.715141+0000", 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0863999999999999e-05 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4136467149999996, 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41242393999999999, 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 110 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.896 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405508+0000", 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3699999999999999e-07 2024-04-26T01:24:00.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405515+0000", 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.669e-06 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405525+0000", 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0657e-05 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405533+0000", 2024-04-26T01:24:00.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0160000000000008e-06 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414083+0000", 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0085497569999999998 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414207+0000", 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012396999999999999 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414400+0000", 2024-04-26T01:24:00.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019254 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520650+0000", 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10625069299999999 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817668+0000", 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29701819000000002 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817820+0000", 2024-04-26T01:24:00.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000151075 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817931+0000", 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011143599999999999 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4136080299999998, 2024-04-26T01:24:00.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51275573399999996, 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 111 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6700000000000003e-07 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405552+0000", 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0930000000000002e-06 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405561+0000", 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5030000000000003e-06 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405572+0000", 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0723000000000001e-05 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414943+0000", 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093715910000000003 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.415187+0000", 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024357900000000001 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.415560+0000", 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00037288799999999998 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520666+0000", 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105105713 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918150+0000", 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.39748471200000002 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918236+0000", 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5780999999999997e-05 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918301+0000", 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5483999999999998e-05 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4135697910000005, 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51318251699999995, 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.907 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 112 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:00.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405585+0000", 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5300000000000003e-07 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405743+0000", 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015815499999999999 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405757+0000", 2024-04-26T01:24:00.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3997e-05 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405767+0000", 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6260000000000007e-06 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416357+0000", 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010590268999999999 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416478+0000", 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012163300000000001 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416669+0000", 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019071999999999999 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522454+0000", 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105784641 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918478+0000", 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.396024563 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918641+0000", 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016244100000000001 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918767+0000", 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012571899999999999 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4133692270000004, 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61300901299999999, 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.913 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 113 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405786+0000", 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5600000000000004e-07 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405803+0000", 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7666e-05 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405814+0000", 2024-04-26T01:24:00.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0747000000000001e-05 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405822+0000", 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8329999999999994e-06 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417320+0000", 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011497772999999999 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417564+0000", 2024-04-26T01:24:00.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000244842 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.418017+0000", 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045273500000000001 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522469+0000", 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104451424 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018558+0000", 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.496089267 2024-04-26T01:24:00.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018655+0000", 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7194000000000005e-05 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018794+0000", 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013857600000000001 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4133189960000001, 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61322511700000004, 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 114 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1699999999999997e-07 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405842+0000", 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8530000000000003e-06 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405852+0000", 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6830000000000003e-06 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405860+0000", 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6159999999999996e-06 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.518995+0000", 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11313486 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519240+0000", 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000245593 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519639+0000", 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039928400000000002 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524261+0000", 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046215249999999996 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018869+0000", 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49460855799999998 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018947+0000", 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7742000000000002e-05 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019060+0000", 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112686 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4132782129999999, 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61349101800000005, 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.924 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:00.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.16e-07 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406016+0000", 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013951499999999999 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406029+0000", 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2819999999999999e-05 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406037+0000", 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6470000000000004e-06 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520774+0000", 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.114736465 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520941+0000", 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016684800000000001 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.521147+0000", 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020602000000000001 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524283+0000", 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003136453 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019192+0000", 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.494908399 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019283+0000", 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1893999999999998e-05 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019367+0000", 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3140999999999996e-05 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4130982330000004, 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61359576599999999, 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6400000000000001e-07 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406062+0000", 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5160000000000002e-06 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406072+0000", 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.91e-06 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406080+0000", 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7030000000000006e-06 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522858+0000", 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116778593 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523093+0000", 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000234763 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523479+0000", 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000385831 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525176+0000", 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016968619999999999 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019468+0000", 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49429284899999998 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019546+0000", 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7553999999999996e-05 2024-04-26T01:24:00.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019651+0000", 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000105321 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 4.4130623069999997, 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61381219799999998, 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:00.935 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 117 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:00.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406092+0000", 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6700000000000003e-07 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406100+0000", 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4560000000000003e-06 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:00.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406109+0000", 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4590000000000004e-06 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.506278+0000", 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10016914 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524347+0000", 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018068623999999998 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524495+0000", 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148276 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524702+0000", 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020681900000000001 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525680+0000", 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00097787599999999992 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019759+0000", 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49407962300000002 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019840+0000", 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0563999999999996e-05 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019904+0000", 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3694000000000004e-05 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.940 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:00.941 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:00.941 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:00.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:00.943 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:00.944+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:00.976 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-04-26T01:24:00.976 INFO:teuthology.orchestra.run.smithi037.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-04-26T01:24:01.059 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:01.064+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:01.082 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:01.082 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:01.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.082 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:190 1.0 e31/23)", 2024-04-26T01:24:01.082 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:01.002616+0000", 2024-04-26T01:24:01.082 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.085599073999999997, 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.085635733000000006, 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:01.002616+0000", 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:01.002616+0000", 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:01.002617+0000", 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0330000000000001e-06 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:01.002685+0000", 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7875999999999999e-05 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:01.002699+0000", 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4156e-05 2024-04-26T01:24:01.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:01.002726+0000", 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6488000000000001e-05 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 1 2024-04-26T01:24:01.085 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:01.097 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:01.144 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:01.148+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:01.218 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:01.219 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:01.219 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:01.219 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:01.219 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:01.233 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:01.245 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:01.244+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:01.346 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:01.348+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:01.361 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:01.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:84 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'54, mlcod=27'50)", 2024-04-26T01:24:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:55.035147+0000", 2024-04-26T01:24:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.3289606200000001, 2024-04-26T01:24:01.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30383523099999998, 2024-04-26T01:24:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.035147+0000", 2024-04-26T01:24:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.035147+0000", 2024-04-26T01:24:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.035148+0000", 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.035205+0000", 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6450999999999999e-05 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.035233+0000", 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8076e-05 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135325+0000", 2024-04-26T01:24:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100092441 2024-04-26T01:24:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.235557+0000", 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10023159099999999 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.235590+0000", 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3482000000000002e-05 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.338918+0000", 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103328025 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.338982+0000", 2024-04-26T01:24:01.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3428000000000003e-05 2024-04-26T01:24:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:85 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'55, mlcod=27'50)", 2024-04-26T01:24:01.370 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:55.135364+0000", 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.228742886, 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20369996100000001, 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135364+0000", 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135364+0000", 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135366+0000", 2024-04-26T01:24:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2500000000000001e-06 2024-04-26T01:24:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135381+0000", 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5739e-05 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135407+0000", 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5225999999999999e-05 2024-04-26T01:24:01.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135417+0000", 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0549e-05 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.236179+0000", 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100761709 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.236236+0000", 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6910999999999998e-05 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.338986+0000", 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10274989599999999 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.339064+0000", 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8681000000000001e-05 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.377 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:86 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'56, mlcod=27'50)", 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:55.135437+0000", 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.2286705959999997, 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20375911299999999, 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135437+0000", 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135437+0000", 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135437+0000", 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.44e-07 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135443+0000", 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8490000000000002e-06 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135453+0000", 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0094e-05 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135460+0000", 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9800000000000001e-06 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.336824+0000", 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20136388899999999 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.336857+0000", 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3071999999999998e-05 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.339067+0000", 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002210016 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.339196+0000", 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012856900000000001 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:87 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'57, mlcod=27'50)", 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:55.135499+0000", 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.2286082179999998, 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20392716999999999, 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135499+0000", 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135499+0000", 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135500+0000", 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7700000000000004e-07 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135578+0000", 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8504000000000006e-05 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135594+0000", 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5716e-05 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135605+0000", 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0613e-05 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.337344+0000", 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20173968 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.337373+0000", 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8552e-05 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.339355+0000", 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019822009999999998 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.339426+0000", 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1227000000000001e-05 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.385 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:88 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'58, mlcod=27'50)", 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:55.135618+0000", 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.2284887290000004, 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.204563575, 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135618+0000", 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135618+0000", 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135619+0000", 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3000000000000001e-07 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135625+0000", 2024-04-26T01:24:01.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6439999999999997e-06 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135636+0000", 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1589e-05 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.135643+0000", 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.37e-06 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.338431+0000", 2024-04-26T01:24:01.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20278842599999999 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.338518+0000", 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7523999999999998e-05 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.340066+0000", 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015473559999999999 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:55.340182+0000", 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000116236 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:107 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'69, mlcod=27'64)", 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.510461+0000", 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8536461329999998, 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20334754599999999, 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.390 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510461+0000", 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510461+0000", 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510463+0000", 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8870000000000001e-06 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510475+0000", 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1873e-05 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510506+0000", 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0828999999999999e-05 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510525+0000", 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9392e-05 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.610755+0000", 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100230021 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.610789+0000", 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3615999999999998e-05 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713730+0000", 2024-04-26T01:24:01.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102941624 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713809+0000", 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8304000000000001e-05 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:108 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'70, mlcod=27'65)", 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.510572+0000", 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8535351530000002, 2024-04-26T01:24:01.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20329510100000001, 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510572+0000", 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510572+0000", 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510573+0000", 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0130000000000001e-06 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510591+0000", 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8023e-05 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510609+0000", 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.783e-05 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510625+0000", 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6475000000000001e-05 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.611276+0000", 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100650163 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.611359+0000", 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3307999999999996e-05 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713814+0000", 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10245544400000001 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713867+0000", 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2845e-05 2024-04-26T01:24:01.398 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:109 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'71, mlcod=27'65)", 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.510650+0000", 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8534575760000003, 2024-04-26T01:24:01.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20441276999999999, 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510650+0000", 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510650+0000", 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510651+0000", 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:24:01.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510722+0000", 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1161000000000005e-05 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510737+0000", 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5403000000000001e-05 2024-04-26T01:24:01.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510752+0000", 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5235999999999999e-05 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.712163+0000", 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20141019800000001 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.712195+0000", 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2369000000000001e-05 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.714985+0000", 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002789826 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.715062+0000", 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7626999999999994e-05 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:110 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'72, mlcod=27'65)", 2024-04-26T01:24:01.404 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.510774+0000", 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.853333557, 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.306793749, 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510774+0000", 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510774+0000", 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510774+0000", 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0999999999999997e-07 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510783+0000", 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5699999999999993e-06 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510803+0000", 2024-04-26T01:24:01.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0098999999999998e-05 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510816+0000", 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2747e-05 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.712766+0000", 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201949726 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.712800+0000", 2024-04-26T01:24:01.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4665000000000001e-05 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817496+0000", 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10469608499999999 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817567+0000", 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1046999999999999e-05 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.408 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:111 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'73, mlcod=27'65)", 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.510834+0000", 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8532732420000002, 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30681685800000003, 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510834+0000", 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510834+0000", 2024-04-26T01:24:01.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510835+0000", 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3300000000000001e-07 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510843+0000", 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5939999999999994e-06 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510865+0000", 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2164999999999999e-05 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510879+0000", 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.308e-05 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713262+0000", 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20238356499999999 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713294+0000", 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2295999999999997e-05 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817575+0000", 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104280284 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817651+0000", 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6141000000000002e-05 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:112 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'74, mlcod=27'65)", 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.510900+0000", 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8532073540000003, 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.306834734, 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510900+0000", 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510900+0000", 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510900+0000", 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6300000000000005e-07 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510967+0000", 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6890999999999994e-05 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510984+0000", 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6322999999999999e-05 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.510997+0000", 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3040000000000001e-05 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713938+0000", 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20294103899999999 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.713982+0000", 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.4412000000000001e-05 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817654+0000", 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103672132 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817735+0000", 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0333999999999994e-05 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:113 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'75, mlcod=27'65)", 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.511016+0000", 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.853091365, 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30679922900000001, 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.511016+0000", 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.511016+0000", 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.511017+0000", 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5700000000000001e-07 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.511025+0000", 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6600000000000001e-06 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.511045+0000", 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9395999999999999e-05 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.511057+0000", 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2533e-05 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.714477+0000", 2024-04-26T01:24:01.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20341987 2024-04-26T01:24:01.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.714509+0000", 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1365999999999998e-05 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817738+0000", 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10322920300000001 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817815+0000", 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7343999999999997e-05 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:114 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'76, mlcod=27'65)", 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.520366+0000", 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8437408729999998, 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29753953700000002, 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.520366+0000", 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.520366+0000", 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.520368+0000", 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2890000000000001e-06 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.520376+0000", 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.867e-06 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.520397+0000", 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0219000000000001e-05 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.520411+0000", 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4646e-05 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.715141+0000", 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19472942700000001 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.715170+0000", 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9699000000000001e-05 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817819+0000", 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10264896 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817906+0000", 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6429999999999997e-05 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.425 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:115 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'77, mlcod=27'65)", 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.521955+0000", 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8421523539999995, 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29601639600000001, 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.521955+0000", 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.521955+0000", 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.521957+0000", 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.807e-06 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.522012+0000", 2024-04-26T01:24:01.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5557999999999999e-05 2024-04-26T01:24:01.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.522042+0000", 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9759000000000001e-05 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.522065+0000", 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2938999999999998e-05 2024-04-26T01:24:01.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.715585+0000", 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19351991800000001 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.715614+0000", 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8952e-05 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817909+0000", 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102295015 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817971+0000", 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2447999999999996e-05 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.432 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:116 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'78, mlcod=27'67)", 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.524220+0000", 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8398870829999998, 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29383424499999999, 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.524220+0000", 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.524220+0000", 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.524222+0000", 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8029999999999999e-06 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.524234+0000", 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1569999999999999e-05 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.524263+0000", 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9731999999999999e-05 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.524282+0000", 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9043e-05 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.816645+0000", 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29236295499999998 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.816678+0000", 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2962000000000001e-05 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817976+0000", 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012977170000000001 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.818054+0000", 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8462999999999997e-05 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.436 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:117 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'79, mlcod=27'67)", 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:23:56.525414+0000", 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 4.8386930399999999, 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29325492199999997, 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.525414+0000", 2024-04-26T01:24:01.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.525414+0000", 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.525416+0000", 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.79e-06 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.525429+0000", 2024-04-26T01:24:01.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3009e-05 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.525458+0000", 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9379000000000001e-05 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.525483+0000", 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5018999999999999e-05 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817166+0000", 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29168264199999999 2024-04-26T01:24:01.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.817217+0000", 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0911000000000001e-05 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.818611+0000", 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00139434 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:23:56.818669+0000", 2024-04-26T01:24:01.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7831999999999998e-05 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:178 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'128, mlcod=30'125)", 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:00.690534+0000", 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.67357367999999995, 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20440318099999999, 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.690534+0000", 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.690534+0000", 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.690535+0000", 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.8299999999999995e-07 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.690543+0000", 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2830000000000001e-06 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.690557+0000", 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4209999999999999e-05 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.690567+0000", 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7759999999999992e-06 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691352+0000", 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000785303 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691387+0000", 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4591999999999997e-05 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.894849+0000", 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20346229399999999 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.894937+0000", 2024-04-26T01:24:01.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7739999999999997e-05 2024-04-26T01:24:01.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.445 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.445 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.446 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:179 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'129, mlcod=30'125)", 2024-04-26T01:24:01.446 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:00.691114+0000", 2024-04-26T01:24:01.446 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.67299277099999999, 2024-04-26T01:24:01.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.203862021, 2024-04-26T01:24:01.446 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.446 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.447 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.447 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691114+0000", 2024-04-26T01:24:01.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691114+0000", 2024-04-26T01:24:01.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691116+0000", 2024-04-26T01:24:01.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8029999999999999e-06 2024-04-26T01:24:01.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691177+0000", 2024-04-26T01:24:01.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0911e-05 2024-04-26T01:24:01.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.450 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.450 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691201+0000", 2024-04-26T01:24:01.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4213e-05 2024-04-26T01:24:01.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691218+0000", 2024-04-26T01:24:01.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6453e-05 2024-04-26T01:24:01.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792040+0000", 2024-04-26T01:24:01.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10082256000000001 2024-04-26T01:24:01.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.452 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.452 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792078+0000", 2024-04-26T01:24:01.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8098e-05 2024-04-26T01:24:01.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.894940+0000", 2024-04-26T01:24:01.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102861814 2024-04-26T01:24:01.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.894976+0000", 2024-04-26T01:24:01.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6168999999999999e-05 2024-04-26T01:24:01.454 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.455 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.455 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.455 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:180 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'130, mlcod=30'125)", 2024-04-26T01:24:01.455 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:00.691627+0000", 2024-04-26T01:24:01.455 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.67247976099999995, 2024-04-26T01:24:01.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20338684700000001, 2024-04-26T01:24:01.456 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.456 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.456 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.456 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691627+0000", 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691627+0000", 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691629+0000", 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6899999999999999e-06 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691644+0000", 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4517e-05 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.691670+0000", 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6772e-05 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.791848+0000", 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10017709599999999 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792642+0000", 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000793979 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792670+0000", 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8002999999999999e-05 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.894981+0000", 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102311981 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.895014+0000", 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2808999999999997e-05 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_repop(client.4245.0:189 1.0 e30/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 30'139, mlcod=30'125)", 2024-04-26T01:24:01.461 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:00.792832+0000", 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.57127471600000002, 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20636190400000001, 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792832+0000", 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.462 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792832+0000", 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792833+0000", 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8299999999999995e-07 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792841+0000", 2024-04-26T01:24:01.463 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0569999999999993e-06 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792857+0000", 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5800000000000001e-05 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.792869+0000", 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1826e-05 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.998168+0000", 2024-04-26T01:24:01.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.205299286 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.998205+0000", 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6829999999999998e-05 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.999145+0000", 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093933500000000002 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:00.999194+0000", 2024-04-26T01:24:01.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9886999999999999e-05 2024-04-26T01:24:01.466 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.466 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.466 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.466 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:01.466 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:01.466 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:01.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:01.468 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:01.468+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:01.568 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:01.572+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:01.640 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:01.640 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:01.641 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:01.641 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:01.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:01.696 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:01.700+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:01.771 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:01.774+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:01.817 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:01.817 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:01.817 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:01.817 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:01.817 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:01.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:01.872 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:01.872+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:01.953 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:01.953 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:01.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:01.953 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [] 2024-04-26T01:24:01.953 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:01.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:02.012 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:02.016+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:02.015 DEBUG:teuthology.orchestra.run.smithi181:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight read -> 100 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight delete -> 50 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight snap_create -> 50 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight snap_remove -> 50 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight rollback -> 50 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight setattr -> 25 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight rmattr -> 25 2024-04-26T01:24:02.037 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight copy_from -> 50 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight append -> 100 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout:ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev) 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout:Configuration: 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout: Number of operations: 400 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout: Number of objects: 50 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout: Max in flight operations: 16 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout: Object size (in bytes): 4000000 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout: Write stride min: 400000 2024-04-26T01:24:02.038 INFO:tasks.rados.rados.0.smithi181.stdout: Write stride max: 800000 2024-04-26T01:24:02.074 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:02.076+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:02.098 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:02.098 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:02.098 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:02.098 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:02.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:02.175 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:02.180+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:02.196 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:02.276 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:02.276+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:02.280 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:02.281 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:02.281 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:02.281 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:02.281 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:02.294 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:02.344 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:02.344 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:02.366 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:02.366 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:02.377 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:02.380+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:02.384 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:02.384 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:02.407 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:02.407 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:02.428 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:02.428 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:02.451 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:02.451 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:02.469 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:02.469 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:02.478 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:02.478+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:02.490 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:02.490 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:02.512 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:02.513 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:02.516 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:02.516 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:02.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:02.516 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [] 2024-04-26T01:24:02.516 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:02.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:02.535 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:02.535 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:02.556 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:02.556 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:02.580 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:02.584+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:02.640 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:02.640 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:02.653 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:02.653 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:02.654 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:02.654 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:02.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:02.681 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:02.684+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:02.756 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:02.756 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:02.782 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:02.784+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:02.835 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:02.835 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:02.835 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:02.836 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:02.836 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:02.849 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:02.882 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:02.882+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:02.896 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:02.896 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:02.979 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [7] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:24:02.980 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:24:02.980 INFO:tasks.thrashosds.thrasher:Removing osd 12, in_osds are: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15] 2024-04-26T01:24:02.980 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 12 2024-04-26T01:24:02.984 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:02.984+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:02.995 INFO:tasks.rados.rados.0.smithi181.stdout:1: write initial oid 1 2024-04-26T01:24:02.995 INFO:tasks.rados.rados.0.smithi181.stdout:1: seq_num 0 ranges {0=32768} 2024-04-26T01:24:02.996 INFO:tasks.rados.rados.0.smithi181.stdout:1: writing smithi18128752-1 from 0 to 32768 tid 1 2024-04-26T01:24:02.996 INFO:tasks.rados.rados.0.smithi181.stdout:2: write initial oid 2 2024-04-26T01:24:02.997 INFO:tasks.rados.rados.0.smithi181.stdout:2: seq_num 1 ranges {0=32768} 2024-04-26T01:24:02.997 INFO:tasks.rados.rados.0.smithi181.stdout:2: writing smithi18128752-2 from 0 to 32768 tid 1 2024-04-26T01:24:02.997 INFO:tasks.rados.rados.0.smithi181.stdout:3: write initial oid 3 2024-04-26T01:24:02.997 INFO:tasks.rados.rados.0.smithi181.stdout:3: seq_num 2 ranges {0=49152} 2024-04-26T01:24:02.997 INFO:tasks.rados.rados.0.smithi181.stdout:3: writing smithi18128752-3 from 0 to 49152 tid 1 2024-04-26T01:24:02.997 INFO:tasks.rados.rados.0.smithi181.stdout:4: write initial oid 4 2024-04-26T01:24:02.997 INFO:tasks.rados.rados.0.smithi181.stdout:4: seq_num 3 ranges {0=65536} 2024-04-26T01:24:02.998 INFO:tasks.rados.rados.0.smithi181.stdout:4: writing smithi18128752-4 from 0 to 65536 tid 1 2024-04-26T01:24:02.998 INFO:tasks.rados.rados.0.smithi181.stdout:5: write initial oid 5 2024-04-26T01:24:02.998 INFO:tasks.rados.rados.0.smithi181.stdout:5: seq_num 4 ranges {0=81920} 2024-04-26T01:24:02.999 INFO:tasks.rados.rados.0.smithi181.stdout:5: writing smithi18128752-5 from 0 to 81920 tid 1 2024-04-26T01:24:02.999 INFO:tasks.rados.rados.0.smithi181.stdout:6: write initial oid 6 2024-04-26T01:24:02.999 INFO:tasks.rados.rados.0.smithi181.stdout:6: seq_num 5 ranges {0=98304} 2024-04-26T01:24:03.000 INFO:tasks.rados.rados.0.smithi181.stdout:6: writing smithi18128752-6 from 0 to 98304 tid 1 2024-04-26T01:24:03.000 INFO:tasks.rados.rados.0.smithi181.stdout:7: write initial oid 7 2024-04-26T01:24:03.000 INFO:tasks.rados.rados.0.smithi181.stdout:7: seq_num 6 ranges {0=114688} 2024-04-26T01:24:03.001 INFO:tasks.rados.rados.0.smithi181.stdout:7: writing smithi18128752-7 from 0 to 114688 tid 1 2024-04-26T01:24:03.001 INFO:tasks.rados.rados.0.smithi181.stdout:8: write initial oid 8 2024-04-26T01:24:03.001 INFO:tasks.rados.rados.0.smithi181.stdout:8: seq_num 7 ranges {0=131072} 2024-04-26T01:24:03.002 INFO:tasks.rados.rados.0.smithi181.stdout:8: writing smithi18128752-8 from 0 to 131072 tid 1 2024-04-26T01:24:03.002 INFO:tasks.rados.rados.0.smithi181.stdout:9: write initial oid 9 2024-04-26T01:24:03.002 INFO:tasks.rados.rados.0.smithi181.stdout:9: seq_num 8 ranges {0=147456} 2024-04-26T01:24:03.004 INFO:tasks.rados.rados.0.smithi181.stdout:9: writing smithi18128752-9 from 0 to 147456 tid 1 2024-04-26T01:24:03.004 INFO:tasks.rados.rados.0.smithi181.stdout:10: write initial oid 10 2024-04-26T01:24:03.004 INFO:tasks.rados.rados.0.smithi181.stdout:10: seq_num 9 ranges {0=163840} 2024-04-26T01:24:03.005 INFO:tasks.rados.rados.0.smithi181.stdout:10: writing smithi18128752-10 from 0 to 163840 tid 1 2024-04-26T01:24:03.005 INFO:tasks.rados.rados.0.smithi181.stdout:11: write initial oid 11 2024-04-26T01:24:03.006 INFO:tasks.rados.rados.0.smithi181.stdout:11: seq_num 10 ranges {0=180224} 2024-04-26T01:24:03.007 INFO:tasks.rados.rados.0.smithi181.stdout:11: writing smithi18128752-11 from 0 to 180224 tid 1 2024-04-26T01:24:03.007 INFO:tasks.rados.rados.0.smithi181.stdout:12: write initial oid 12 2024-04-26T01:24:03.007 INFO:tasks.rados.rados.0.smithi181.stdout:12: seq_num 11 ranges {0=196608} 2024-04-26T01:24:03.009 INFO:tasks.rados.rados.0.smithi181.stdout:12: writing smithi18128752-12 from 0 to 196608 tid 1 2024-04-26T01:24:03.009 INFO:tasks.rados.rados.0.smithi181.stdout:13: write initial oid 13 2024-04-26T01:24:03.009 INFO:tasks.rados.rados.0.smithi181.stdout:13: seq_num 12 ranges {0=212992} 2024-04-26T01:24:03.011 INFO:tasks.rados.rados.0.smithi181.stdout:13: writing smithi18128752-13 from 0 to 212992 tid 1 2024-04-26T01:24:03.011 INFO:tasks.rados.rados.0.smithi181.stdout:14: write initial oid 14 2024-04-26T01:24:03.012 INFO:tasks.rados.rados.0.smithi181.stdout:14: seq_num 13 ranges {0=229376} 2024-04-26T01:24:03.013 INFO:tasks.rados.rados.0.smithi181.stdout:14: writing smithi18128752-14 from 0 to 229376 tid 1 2024-04-26T01:24:03.013 INFO:tasks.rados.rados.0.smithi181.stdout:15: write initial oid 15 2024-04-26T01:24:03.013 INFO:tasks.rados.rados.0.smithi181.stdout:15: seq_num 14 ranges {0=245760} 2024-04-26T01:24:03.015 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:24:03.016 INFO:tasks.rados.rados.0.smithi181.stdout:15: writing smithi18128752-15 from 0 to 245760 tid 1 2024-04-26T01:24:03.016 INFO:tasks.rados.rados.0.smithi181.stdout:16: write initial oid 16 2024-04-26T01:24:03.016 INFO:tasks.rados.rados.0.smithi181.stdout:16: seq_num 15 ranges {0=262144} 2024-04-26T01:24:03.018 INFO:tasks.rados.rados.0.smithi181.stdout:16: writing smithi18128752-16 from 0 to 262144 tid 1 2024-04-26T01:24:03.018 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.019 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:03.019 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:03.019 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:03.019 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [] 2024-04-26T01:24:03.019 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:03.024 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:03.025 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:03.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:03.085 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:03.086+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:03.101 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:03.101 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:03.164 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:03.164 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:03.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:03.186 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:03.188+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:03.223 INFO:tasks.rados.rados.0.smithi181.stdout:1: finishing write tid 1 to smithi18128752-1 2024-04-26T01:24:03.287 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:03.288+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:03.321 INFO:tasks.rados.rados.0.smithi181.stdout:7: finishing write tid 1 to smithi18128752-7 2024-04-26T01:24:03.322 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T01:24:03.324+0000 7f533fe62640 -1 mon.a@0(leader).osd e33 definitely_dead 0 2024-04-26T01:24:03.341 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:03.341 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:03.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:03.387 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:03.388+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:03.411 INFO:tasks.rados.rados.0.smithi181.stdout:1: finishing write tid 2 to smithi18128752-1 2024-04-26T01:24:03.412 INFO:tasks.rados.rados.0.smithi181.stdout:11: finishing write tid 1 to smithi18128752-11 2024-04-26T01:24:03.413 INFO:tasks.rados.rados.0.smithi181.stdout:9: finishing write tid 1 to smithi18128752-9 2024-04-26T01:24:03.413 INFO:tasks.rados.rados.0.smithi181.stdout:9: finishing write tid 2 to smithi18128752-9 2024-04-26T01:24:03.419 INFO:tasks.rados.rados.0.smithi181.stdout:12: finishing write tid 1 to smithi18128752-12 2024-04-26T01:24:03.422 INFO:tasks.rados.rados.0.smithi181.stdout:7: finishing write tid 2 to smithi18128752-7 2024-04-26T01:24:03.488 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:03.492+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:03.505 INFO:tasks.rados.rados.0.smithi181.stdout:3: finishing write tid 1 to smithi18128752-3 2024-04-26T01:24:03.513 INFO:tasks.rados.rados.0.smithi181.stdout:11: finishing write tid 2 to smithi18128752-11 2024-04-26T01:24:03.519 INFO:tasks.rados.rados.0.smithi181.stdout:12: finishing write tid 2 to smithi18128752-12 2024-04-26T01:24:03.525 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:03.526 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:03.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:03.589 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:03.592+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:03.605 INFO:tasks.rados.rados.0.smithi181.stdout:3: finishing write tid 2 to smithi18128752-3 2024-04-26T01:24:03.614 INFO:tasks.rados.rados.0.smithi181.stdout:16: finishing write tid 1 to smithi18128752-16 2024-04-26T01:24:03.614 INFO:tasks.rados.rados.0.smithi181.stdout:15: finishing write tid 1 to smithi18128752-15 2024-04-26T01:24:03.614 INFO:tasks.rados.rados.0.smithi181.stdout:15: finishing write tid 2 to smithi18128752-15 2024-04-26T01:24:03.616 INFO:tasks.rados.rados.0.smithi181.stdout:14: finishing write tid 1 to smithi18128752-14 2024-04-26T01:24:03.616 INFO:tasks.rados.rados.0.smithi181.stdout:2: finishing write tid 1 to smithi18128752-2 2024-04-26T01:24:03.624 INFO:tasks.rados.rados.0.smithi181.stdout:16: finishing write tid 2 to smithi18128752-16 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:7: finishing write tid 3 to smithi18128752-7 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:7: oid 7 updating version 0 to 1 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:7: oid 7 updating version 1 to 2 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:7: done (15 left) 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:17: write initial oid 17 2024-04-26T01:24:03.627 INFO:tasks.rados.rados.0.smithi181.stdout:17: seq_num 16 ranges {0=278528} 2024-04-26T01:24:03.628 INFO:tasks.rados.rados.0.smithi181.stdout:17: writing smithi18128752-17 from 0 to 278528 tid 1 2024-04-26T01:24:03.629 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.689 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:03.692+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:03.708 INFO:tasks.rados.rados.0.smithi181.stdout:14: finishing write tid 2 to smithi18128752-14 2024-04-26T01:24:03.710 INFO:tasks.rados.rados.0.smithi181.stdout:8: finishing write tid 1 to smithi18128752-8 2024-04-26T01:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:03.711 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:03.711 INFO:tasks.rados.rados.0.smithi181.stdout:8: finishing write tid 2 to smithi18128752-8 2024-04-26T01:24:03.713 INFO:tasks.rados.rados.0.smithi181.stdout:4: finishing write tid 1 to smithi18128752-4 2024-04-26T01:24:03.713 INFO:tasks.rados.rados.0.smithi181.stdout:4: finishing write tid 2 to smithi18128752-4 2024-04-26T01:24:03.713 INFO:tasks.rados.rados.0.smithi181.stdout:6: finishing write tid 1 to smithi18128752-6 2024-04-26T01:24:03.717 INFO:tasks.rados.rados.0.smithi181.stdout:2: finishing write tid 2 to smithi18128752-2 2024-04-26T01:24:03.721 INFO:tasks.rados.rados.0.smithi181.stdout:12: finishing write tid 3 to smithi18128752-12 2024-04-26T01:24:03.721 INFO:tasks.rados.rados.0.smithi181.stdout:12: oid 12 updating version 0 to 1 2024-04-26T01:24:03.721 INFO:tasks.rados.rados.0.smithi181.stdout:12: oid 12 updating version 1 to 2 2024-04-26T01:24:03.721 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-04-26T01:24:03.721 INFO:tasks.rados.rados.0.smithi181.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-04-26T01:24:03.722 INFO:tasks.rados.rados.0.smithi181.stdout:12: done (15 left) 2024-04-26T01:24:03.722 INFO:tasks.rados.rados.0.smithi181.stdout:18: write initial oid 18 2024-04-26T01:24:03.722 INFO:tasks.rados.rados.0.smithi181.stdout:18: seq_num 17 ranges {0=294912} 2024-04-26T01:24:03.724 INFO:tasks.rados.rados.0.smithi181.stdout:18: writing smithi18128752-18 from 0 to 294912 tid 1 2024-04-26T01:24:03.724 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:03.790 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:03.792+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:3: finishing write tid 3 to smithi18128752-3 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:3: oid 3 updating version 0 to 1 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:3: oid 3 updating version 1 to 2 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:3: done (15 left) 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:19: write initial oid 19 2024-04-26T01:24:03.811 INFO:tasks.rados.rados.0.smithi181.stdout:19: seq_num 18 ranges {0=311296} 2024-04-26T01:24:03.813 INFO:tasks.rados.rados.0.smithi181.stdout:19: writing smithi18128752-19 from 0 to 311296 tid 1 2024-04-26T01:24:03.813 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.814 INFO:tasks.rados.rados.0.smithi181.stdout:5: finishing write tid 1 to smithi18128752-5 2024-04-26T01:24:03.814 INFO:tasks.rados.rados.0.smithi181.stdout:5: finishing write tid 2 to smithi18128752-5 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:9: finishing write tid 3 to smithi18128752-9 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:9: oid 9 updating version 0 to 2 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:9: oid 9 version 2 is already newer than 1 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:9: done (15 left) 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:20: write initial oid 20 2024-04-26T01:24:03.815 INFO:tasks.rados.rados.0.smithi181.stdout:20: seq_num 19 ranges {0=327680} 2024-04-26T01:24:03.817 INFO:tasks.rados.rados.0.smithi181.stdout:20: writing smithi18128752-20 from 0 to 327680 tid 1 2024-04-26T01:24:03.817 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:11: finishing write tid 3 to smithi18128752-11 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:11: oid 11 updating version 0 to 1 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:11: oid 11 updating version 1 to 2 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:11: done (15 left) 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:21: write initial oid 21 2024-04-26T01:24:03.818 INFO:tasks.rados.rados.0.smithi181.stdout:21: seq_num 20 ranges {0=344064} 2024-04-26T01:24:03.820 INFO:tasks.rados.rados.0.smithi181.stdout:21: writing smithi18128752-21 from 0 to 344064 tid 1 2024-04-26T01:24:03.820 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.821 INFO:tasks.rados.rados.0.smithi181.stdout:15: finishing write tid 3 to smithi18128752-15 2024-04-26T01:24:03.822 INFO:tasks.rados.rados.0.smithi181.stdout:15: oid 15 updating version 0 to 1 2024-04-26T01:24:03.822 INFO:tasks.rados.rados.0.smithi181.stdout:15: oid 15 updating version 1 to 2 2024-04-26T01:24:03.822 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 2 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-04-26T01:24:03.822 INFO:tasks.rados.rados.0.smithi181.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-04-26T01:24:03.822 INFO:tasks.rados.rados.0.smithi181.stdout:15: done (15 left) 2024-04-26T01:24:03.822 INFO:tasks.rados.rados.0.smithi181.stdout:22: write initial oid 22 2024-04-26T01:24:03.822 INFO:tasks.rados.rados.0.smithi181.stdout:22: seq_num 21 ranges {0=360448} 2024-04-26T01:24:03.824 INFO:tasks.rados.rados.0.smithi181.stdout:22: writing smithi18128752-22 from 0 to 360448 tid 1 2024-04-26T01:24:03.824 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.826 INFO:tasks.rados.rados.0.smithi181.stdout:13: finishing write tid 1 to smithi18128752-13 2024-04-26T01:24:03.826 INFO:tasks.rados.rados.0.smithi181.stdout:13: finishing write tid 2 to smithi18128752-13 2024-04-26T01:24:03.826 INFO:tasks.rados.rados.0.smithi181.stdout:10: finishing write tid 1 to smithi18128752-10 2024-04-26T01:24:03.827 INFO:tasks.rados.rados.0.smithi181.stdout:10: finishing write tid 2 to smithi18128752-10 2024-04-26T01:24:03.889 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:03.889 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:03.891 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:03.896+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:03.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:24:03.914 INFO:tasks.rados.rados.0.smithi181.stdout:6: finishing write tid 2 to smithi18128752-6 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:16: finishing write tid 3 to smithi18128752-16 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:16: oid 16 updating version 0 to 3 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:16: oid 16 updating version 3 to 4 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 4 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:16: done (15 left) 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:23: write initial oid 23 2024-04-26T01:24:03.920 INFO:tasks.rados.rados.0.smithi181.stdout:23: seq_num 22 ranges {0=376832} 2024-04-26T01:24:03.922 INFO:tasks.rados.rados.0.smithi181.stdout:23: writing smithi18128752-23 from 0 to 376832 tid 1 2024-04-26T01:24:03.922 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:03.992 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:03.996+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:04.002 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:24:04.010 INFO:teuthology.orchestra.run.smithi037.stderr:marked out osd.12. 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:8: finishing write tid 3 to smithi18128752-8 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:8: oid 8 updating version 0 to 1 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:8: oid 8 updating version 1 to 2 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:8: done (15 left) 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:24: write initial oid 24 2024-04-26T01:24:04.016 INFO:tasks.rados.rados.0.smithi181.stdout:24: seq_num 23 ranges {0=393216} 2024-04-26T01:24:04.023 INFO:tasks.rados.rados.0.smithi181.stdout:24: writing smithi18128752-24 from 0 to 393216 tid 1 2024-04-26T01:24:04.023 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.040 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:04.040 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:04.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:24:04.092 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:04.092+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:04.117 INFO:tasks.rados.rados.0.smithi181.stdout:6: finishing write tid 3 to smithi18128752-6 2024-04-26T01:24:04.118 INFO:tasks.rados.rados.0.smithi181.stdout:6: oid 6 updating version 0 to 1 2024-04-26T01:24:04.118 INFO:tasks.rados.rados.0.smithi181.stdout:6: oid 6 updating version 1 to 2 2024-04-26T01:24:04.118 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-04-26T01:24:04.118 INFO:tasks.rados.rados.0.smithi181.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-04-26T01:24:04.118 INFO:tasks.rados.rados.0.smithi181.stdout:6: done (15 left) 2024-04-26T01:24:04.118 INFO:tasks.rados.rados.0.smithi181.stdout:25: write initial oid 25 2024-04-26T01:24:04.118 INFO:tasks.rados.rados.0.smithi181.stdout:25: seq_num 24 ranges {0=409600} 2024-04-26T01:24:04.120 INFO:tasks.rados.rados.0.smithi181.stdout:25: writing smithi18128752-25 from 0 to 409600 tid 1 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:14: finishing write tid 3 to smithi18128752-14 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:14: oid 14 updating version 0 to 1 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:14: oid 14 updating version 1 to 2 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 14 v 2 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:14: done (15 left) 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:26: write initial oid 26 2024-04-26T01:24:04.121 INFO:tasks.rados.rados.0.smithi181.stdout:26: seq_num 25 ranges {0=425984} 2024-04-26T01:24:04.124 INFO:tasks.rados.rados.0.smithi181.stdout:26: writing smithi18128752-26 from 0 to 425984 tid 1 2024-04-26T01:24:04.124 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.125 INFO:tasks.rados.rados.0.smithi181.stdout:2: finishing write tid 3 to smithi18128752-2 2024-04-26T01:24:04.125 INFO:tasks.rados.rados.0.smithi181.stdout:2: oid 2 updating version 0 to 1 2024-04-26T01:24:04.125 INFO:tasks.rados.rados.0.smithi181.stdout:2: oid 2 updating version 1 to 2 2024-04-26T01:24:04.125 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-04-26T01:24:04.126 INFO:tasks.rados.rados.0.smithi181.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-04-26T01:24:04.126 INFO:tasks.rados.rados.0.smithi181.stdout:2: done (15 left) 2024-04-26T01:24:04.126 INFO:tasks.rados.rados.0.smithi181.stdout:27: write initial oid 27 2024-04-26T01:24:04.126 INFO:tasks.rados.rados.0.smithi181.stdout:27: seq_num 26 ranges {0=442368} 2024-04-26T01:24:04.129 INFO:tasks.rados.rados.0.smithi181.stdout:27: writing smithi18128752-27 from 0 to 442368 tid 1 2024-04-26T01:24:04.129 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.129 INFO:tasks.rados.rados.0.smithi181.stdout:10: finishing write tid 3 to smithi18128752-10 2024-04-26T01:24:04.129 INFO:tasks.rados.rados.0.smithi181.stdout:10: oid 10 updating version 0 to 3 2024-04-26T01:24:04.129 INFO:tasks.rados.rados.0.smithi181.stdout:10: oid 10 updating version 3 to 4 2024-04-26T01:24:04.129 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 4 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-26T01:24:04.129 INFO:tasks.rados.rados.0.smithi181.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-04-26T01:24:04.130 INFO:tasks.rados.rados.0.smithi181.stdout:10: done (15 left) 2024-04-26T01:24:04.130 INFO:tasks.rados.rados.0.smithi181.stdout:28: write initial oid 28 2024-04-26T01:24:04.130 INFO:tasks.rados.rados.0.smithi181.stdout:28: seq_num 27 ranges {0=458752} 2024-04-26T01:24:04.132 INFO:tasks.rados.rados.0.smithi181.stdout:28: writing smithi18128752-28 from 0 to 458752 tid 1 2024-04-26T01:24:04.133 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.194 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:04.196+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:04.220 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:04.220 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:04.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:24:04.294 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:04.296+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:04.326 INFO:tasks.rados.rados.0.smithi181.stdout:17: finishing write tid 1 to smithi18128752-17 2024-04-26T01:24:04.341 INFO:tasks.rados.rados.0.smithi181.stdout:28: finishing write tid 1 to smithi18128752-28 2024-04-26T01:24:04.341 INFO:tasks.rados.rados.0.smithi181.stdout:28: finishing write tid 2 to smithi18128752-28 2024-04-26T01:24:04.395 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:04.400+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:04.398 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:04.398 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:04.414 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:24:04.431 INFO:tasks.rados.rados.0.smithi181.stdout:18: finishing write tid 1 to smithi18128752-18 2024-04-26T01:24:04.431 INFO:tasks.rados.rados.0.smithi181.stdout:13: finishing write tid 3 to smithi18128752-13 2024-04-26T01:24:04.431 INFO:tasks.rados.rados.0.smithi181.stdout:13: oid 13 updating version 0 to 4 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:13: oid 13 version 4 is already newer than 3 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:18: finishing write tid 2 to smithi18128752-18 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:19: finishing write tid 1 to smithi18128752-19 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:13: done (15 left) 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:29: write initial oid 29 2024-04-26T01:24:04.432 INFO:tasks.rados.rados.0.smithi181.stdout:29: seq_num 28 ranges {0=475136} 2024-04-26T01:24:04.435 INFO:tasks.rados.rados.0.smithi181.stdout:29: writing smithi18128752-29 from 0 to 475136 tid 1 2024-04-26T01:24:04.435 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.435 INFO:tasks.rados.rados.0.smithi181.stdout:19: finishing write tid 2 to smithi18128752-19 2024-04-26T01:24:04.496 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:04.500+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:04.527 INFO:tasks.rados.rados.0.smithi181.stdout:17: finishing write tid 2 to smithi18128752-17 2024-04-26T01:24:04.536 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:04.537 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:04.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:24:04.596 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:04.600+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:04.629 INFO:tasks.rados.rados.0.smithi181.stdout:28: finishing write tid 3 to smithi18128752-28 2024-04-26T01:24:04.629 INFO:tasks.rados.rados.0.smithi181.stdout:28: oid 28 updating version 0 to 1 2024-04-26T01:24:04.629 INFO:tasks.rados.rados.0.smithi181.stdout:28: oid 28 updating version 1 to 2 2024-04-26T01:24:04.629 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 28 v 2 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-26T01:24:04.630 INFO:tasks.rados.rados.0.smithi181.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:24:04.630 INFO:tasks.rados.rados.0.smithi181.stdout:28: done (15 left) 2024-04-26T01:24:04.630 INFO:tasks.rados.rados.0.smithi181.stdout:30: write initial oid 30 2024-04-26T01:24:04.630 INFO:tasks.rados.rados.0.smithi181.stdout:30: seq_num 29 ranges {0=491520} 2024-04-26T01:24:04.633 INFO:tasks.rados.rados.0.smithi181.stdout:30: writing smithi18128752-30 from 0 to 491520 tid 1 2024-04-26T01:24:04.633 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.634 INFO:tasks.rados.rados.0.smithi181.stdout:27: finishing write tid 1 to smithi18128752-27 2024-04-26T01:24:04.634 INFO:tasks.rados.rados.0.smithi181.stdout:27: finishing write tid 2 to smithi18128752-27 2024-04-26T01:24:04.697 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:04.700+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:04.720 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:04.720 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:04.730 INFO:tasks.rados.rados.0.smithi181.stdout:18: finishing write tid 3 to smithi18128752-18 2024-04-26T01:24:04.730 INFO:tasks.rados.rados.0.smithi181.stdout:18: oid 18 updating version 0 to 6 2024-04-26T01:24:04.731 INFO:tasks.rados.rados.0.smithi181.stdout:18: oid 18 version 6 is already newer than 5 2024-04-26T01:24:04.731 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-26T01:24:04.731 INFO:tasks.rados.rados.0.smithi181.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:24:04.731 INFO:tasks.rados.rados.0.smithi181.stdout:18: done (15 left) 2024-04-26T01:24:04.731 INFO:tasks.rados.rados.0.smithi181.stdout:31: write initial oid 31 2024-04-26T01:24:04.731 INFO:tasks.rados.rados.0.smithi181.stdout:31: seq_num 30 ranges {0=507904} 2024-04-26T01:24:04.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout:31: writing smithi18128752-31 from 0 to 507904 tid 1 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout:1: finishing write tid 3 to smithi18128752-1 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout:1: oid 1 updating version 0 to 2 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout:1: oid 1 version 2 is already newer than 1 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-04-26T01:24:04.740 INFO:tasks.rados.rados.0.smithi181.stdout:1: done (15 left) 2024-04-26T01:24:04.741 INFO:tasks.rados.rados.0.smithi181.stdout:32: write initial oid 32 2024-04-26T01:24:04.741 INFO:tasks.rados.rados.0.smithi181.stdout:32: seq_num 31 ranges {0=524288} 2024-04-26T01:24:04.749 INFO:tasks.rados.rados.0.smithi181.stdout:32: writing smithi18128752-32 from 0 to 524288 tid 1 2024-04-26T01:24:04.749 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.799 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:04.800+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:04.831 INFO:tasks.rados.rados.0.smithi181.stdout:4: finishing write tid 3 to smithi18128752-4 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:4: oid 4 updating version 0 to 3 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:4: oid 4 updating version 3 to 4 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:5: finishing write tid 3 to smithi18128752-5 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:5: oid 5 updating version 0 to 5 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:5: oid 5 updating version 5 to 6 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 5 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:4: done (15 left) 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:5: done (14 left) 2024-04-26T01:24:04.832 INFO:tasks.rados.rados.0.smithi181.stdout:33: write initial oid 33 2024-04-26T01:24:04.833 INFO:tasks.rados.rados.0.smithi181.stdout:33: seq_num 32 ranges {0=540672} 2024-04-26T01:24:04.842 INFO:tasks.rados.rados.0.smithi181.stdout:33: writing smithi18128752-33 from 0 to 540672 tid 1 2024-04-26T01:24:04.842 INFO:tasks.rados.rados.0.smithi181.stdout:34: write initial oid 34 2024-04-26T01:24:04.842 INFO:tasks.rados.rados.0.smithi181.stdout:34: seq_num 33 ranges {0=557056} 2024-04-26T01:24:04.851 INFO:tasks.rados.rados.0.smithi181.stdout:34: writing smithi18128752-34 from 0 to 557056 tid 1 2024-04-26T01:24:04.851 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.851 INFO:tasks.rados.rados.0.smithi181.stdout:30: finishing write tid 1 to smithi18128752-30 2024-04-26T01:24:04.851 INFO:tasks.rados.rados.0.smithi181.stdout:19: finishing write tid 3 to smithi18128752-19 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:19: oid 19 updating version 0 to 5 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:19: oid 19 updating version 5 to 6 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:27: finishing write tid 3 to smithi18128752-27 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:27: oid 27 updating version 0 to 8 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:27: oid 27 version 8 is already newer than 7 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 27 v 8 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:19: done (15 left) 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:27: done (14 left) 2024-04-26T01:24:04.852 INFO:tasks.rados.rados.0.smithi181.stdout:35: write initial oid 35 2024-04-26T01:24:04.853 INFO:tasks.rados.rados.0.smithi181.stdout:35: seq_num 34 ranges {0=573440} 2024-04-26T01:24:04.856 INFO:tasks.rados.rados.0.smithi181.stdout:35: writing smithi18128752-35 from 0 to 573440 tid 1 2024-04-26T01:24:04.856 INFO:tasks.rados.rados.0.smithi181.stdout:36: write initial oid 36 2024-04-26T01:24:04.856 INFO:tasks.rados.rados.0.smithi181.stdout:36: seq_num 35 ranges {0=589824} 2024-04-26T01:24:04.861 INFO:tasks.rados.rados.0.smithi181.stdout:36: writing smithi18128752-36 from 0 to 589824 tid 1 2024-04-26T01:24:04.861 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:04.865 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:04.865 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:04.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:24:04.899 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:04.900+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:04.943 INFO:tasks.rados.rados.0.smithi181.stdout:30: finishing write tid 2 to smithi18128752-30 2024-04-26T01:24:05.000 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:05.002+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:05.045 INFO:tasks.rados.rados.0.smithi181.stdout:31: finishing write tid 1 to smithi18128752-31 2024-04-26T01:24:05.046 INFO:tasks.rados.rados.0.smithi181.stdout:32: finishing write tid 1 to smithi18128752-32 2024-04-26T01:24:05.047 INFO:tasks.rados.rados.0.smithi181.stdout:31: finishing write tid 2 to smithi18128752-31 2024-04-26T01:24:05.053 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:05.053 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:05.068 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:24:05.101 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:05.100+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:05.108 INFO:tasks.rados.rados.0.smithi181.stdout:36: finishing write tid 1 to smithi18128752-36 2024-04-26T01:24:05.133 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:30: finishing write tid 3 to smithi18128752-30 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:30: oid 30 updating version 0 to 3 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:30: oid 30 updating version 3 to 4 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 4 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:30: done (15 left) 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:37: write initial oid 37 2024-04-26T01:24:05.152 INFO:tasks.rados.rados.0.smithi181.stdout:37: seq_num 36 ranges {0=606208} 2024-04-26T01:24:05.160 INFO:tasks.rados.rados.0.smithi181.stdout:37: writing smithi18128752-37 from 0 to 606208 tid 1 2024-04-26T01:24:05.161 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.201 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:05.204+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:05.206 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:05.207 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:05.223 INFO:tasks.rados.rados.0.smithi181.stdout:34: finishing write tid 1 to smithi18128752-34 2024-04-26T01:24:05.224 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:24:05.247 INFO:tasks.rados.rados.0.smithi181.stdout:32: finishing write tid 2 to smithi18128752-32 2024-04-26T01:24:05.259 INFO:tasks.rados.rados.0.smithi181.stdout:31: finishing write tid 3 to smithi18128752-31 2024-04-26T01:24:05.259 INFO:tasks.rados.rados.0.smithi181.stdout:31: oid 31 updating version 0 to 4 2024-04-26T01:24:05.259 INFO:tasks.rados.rados.0.smithi181.stdout:31: oid 31 version 4 is already newer than 3 2024-04-26T01:24:05.259 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 4 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-04-26T01:24:05.259 INFO:tasks.rados.rados.0.smithi181.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-04-26T01:24:05.260 INFO:tasks.rados.rados.0.smithi181.stdout:31: done (15 left) 2024-04-26T01:24:05.260 INFO:tasks.rados.rados.0.smithi181.stdout:38: write initial oid 38 2024-04-26T01:24:05.260 INFO:tasks.rados.rados.0.smithi181.stdout:38: seq_num 37 ranges {0=622592} 2024-04-26T01:24:05.270 INFO:tasks.rados.rados.0.smithi181.stdout:38: writing smithi18128752-38 from 0 to 622592 tid 1 2024-04-26T01:24:05.270 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.271 INFO:tasks.rados.rados.0.smithi181.stdout:35: finishing write tid 1 to smithi18128752-35 2024-04-26T01:24:05.271 INFO:tasks.rados.rados.0.smithi181.stdout:21: finishing write tid 1 to smithi18128752-21 2024-04-26T01:24:05.271 INFO:tasks.rados.rados.0.smithi181.stdout:36: finishing write tid 2 to smithi18128752-36 2024-04-26T01:24:05.290 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:05.291 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:05.302 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:05.304+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:05.311 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:05.311 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:05.333 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:05.334 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:05.355 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:05.355 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:05.368 INFO:tasks.rados.rados.0.smithi181.stdout:21: finishing write tid 2 to smithi18128752-21 2024-04-26T01:24:05.368 INFO:tasks.rados.rados.0.smithi181.stdout:17: finishing write tid 3 to smithi18128752-17 2024-04-26T01:24:05.368 INFO:tasks.rados.rados.0.smithi181.stdout:17: oid 17 updating version 0 to 8 2024-04-26T01:24:05.368 INFO:tasks.rados.rados.0.smithi181.stdout:17: oid 17 version 8 is already newer than 7 2024-04-26T01:24:05.369 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 8 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-04-26T01:24:05.369 INFO:tasks.rados.rados.0.smithi181.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-04-26T01:24:05.369 INFO:tasks.rados.rados.0.smithi181.stdout:37: finishing write tid 1 to smithi18128752-37 2024-04-26T01:24:05.369 INFO:tasks.rados.rados.0.smithi181.stdout:17: done (15 left) 2024-04-26T01:24:05.369 INFO:tasks.rados.rados.0.smithi181.stdout:39: write initial oid 39 2024-04-26T01:24:05.369 INFO:tasks.rados.rados.0.smithi181.stdout:39: seq_num 38 ranges {0=638976} 2024-04-26T01:24:05.380 INFO:tasks.rados.rados.0.smithi181.stdout:39: writing smithi18128752-39 from 0 to 638976 tid 1 2024-04-26T01:24:05.380 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.380 INFO:tasks.rados.rados.0.smithi181.stdout:21: finishing write tid 3 to smithi18128752-21 2024-04-26T01:24:05.380 INFO:tasks.rados.rados.0.smithi181.stdout:21: oid 21 updating version 0 to 10 2024-04-26T01:24:05.380 INFO:tasks.rados.rados.0.smithi181.stdout:21: oid 21 version 10 is already newer than 9 2024-04-26T01:24:05.380 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 21 v 10 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-26T01:24:05.381 INFO:tasks.rados.rados.0.smithi181.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-04-26T01:24:05.381 INFO:tasks.rados.rados.0.smithi181.stdout:21: done (15 left) 2024-04-26T01:24:05.381 INFO:tasks.rados.rados.0.smithi181.stdout:40: write initial oid 40 2024-04-26T01:24:05.381 INFO:tasks.rados.rados.0.smithi181.stdout:40: seq_num 39 ranges {0=671744} 2024-04-26T01:24:05.386 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:05.386 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:05.386 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:05.386 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:05.386 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:05.390 INFO:tasks.rados.rados.0.smithi181.stdout:40: writing smithi18128752-40 from 0 to 671744 tid 1 2024-04-26T01:24:05.391 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.400 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:24:05.403 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:05.404+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:05.424 INFO:tasks.rados.rados.0.smithi181.stdout:34: finishing write tid 2 to smithi18128752-34 2024-04-26T01:24:05.452 INFO:tasks.rados.rados.0.smithi181.stdout:33: finishing write tid 1 to smithi18128752-33 2024-04-26T01:24:05.452 INFO:tasks.rados.rados.0.smithi181.stdout:33: finishing write tid 2 to smithi18128752-33 2024-04-26T01:24:05.466 INFO:tasks.rados.rados.0.smithi181.stdout:35: finishing write tid 2 to smithi18128752-35 2024-04-26T01:24:05.466 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:05.466 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:05.474 INFO:tasks.rados.rados.0.smithi181.stdout:36: finishing write tid 3 to smithi18128752-36 2024-04-26T01:24:05.474 INFO:tasks.rados.rados.0.smithi181.stdout:36: oid 36 updating version 0 to 4 2024-04-26T01:24:05.475 INFO:tasks.rados.rados.0.smithi181.stdout:36: oid 36 version 4 is already newer than 3 2024-04-26T01:24:05.475 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 4 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-26T01:24:05.475 INFO:tasks.rados.rados.0.smithi181.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-04-26T01:24:05.475 INFO:tasks.rados.rados.0.smithi181.stdout:37: finishing write tid 2 to smithi18128752-37 2024-04-26T01:24:05.475 INFO:tasks.rados.rados.0.smithi181.stdout:36: done (15 left) 2024-04-26T01:24:05.475 INFO:tasks.rados.rados.0.smithi181.stdout:41: write initial oid 41 2024-04-26T01:24:05.475 INFO:tasks.rados.rados.0.smithi181.stdout:41: seq_num 40 ranges {0=688128} 2024-04-26T01:24:05.486 INFO:tasks.rados.rados.0.smithi181.stdout:41: writing smithi18128752-41 from 0 to 688128 tid 1 2024-04-26T01:24:05.486 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.497 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:05.497 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:05.504 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:05.508+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:05.526 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:05.527 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:05.528 INFO:tasks.rados.rados.0.smithi181.stdout:34: finishing write tid 3 to smithi18128752-34 2024-04-26T01:24:05.528 INFO:tasks.rados.rados.0.smithi181.stdout:34: oid 34 updating version 0 to 6 2024-04-26T01:24:05.528 INFO:tasks.rados.rados.0.smithi181.stdout:34: oid 34 version 6 is already newer than 5 2024-04-26T01:24:05.529 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-04-26T01:24:05.529 INFO:tasks.rados.rados.0.smithi181.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-04-26T01:24:05.529 INFO:tasks.rados.rados.0.smithi181.stdout:34: done (15 left) 2024-04-26T01:24:05.529 INFO:tasks.rados.rados.0.smithi181.stdout:42: write initial oid 42 2024-04-26T01:24:05.529 INFO:tasks.rados.rados.0.smithi181.stdout:42: seq_num 41 ranges {0=704512} 2024-04-26T01:24:05.539 INFO:tasks.rados.rados.0.smithi181.stdout:42: writing smithi18128752-42 from 0 to 704512 tid 1 2024-04-26T01:24:05.539 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:32: finishing write tid 3 to smithi18128752-32 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:32: oid 32 updating version 0 to 10 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:32: oid 32 version 10 is already newer than 9 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 32 v 10 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:32: done (15 left) 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:43: write initial oid 43 2024-04-26T01:24:05.554 INFO:tasks.rados.rados.0.smithi181.stdout:43: seq_num 42 ranges {0=720896} 2024-04-26T01:24:05.555 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:05.555 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:05.567 INFO:tasks.rados.rados.0.smithi181.stdout:43: writing smithi18128752-43 from 0 to 720896 tid 1 2024-04-26T01:24:05.567 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.567 INFO:tasks.rados.rados.0.smithi181.stdout:33: finishing write tid 3 to smithi18128752-33 2024-04-26T01:24:05.567 INFO:tasks.rados.rados.0.smithi181.stdout:33: oid 33 updating version 0 to 6 2024-04-26T01:24:05.567 INFO:tasks.rados.rados.0.smithi181.stdout:33: oid 33 version 6 is already newer than 5 2024-04-26T01:24:05.568 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-26T01:24:05.568 INFO:tasks.rados.rados.0.smithi181.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:05.568 INFO:tasks.rados.rados.0.smithi181.stdout:33: done (15 left) 2024-04-26T01:24:05.568 INFO:tasks.rados.rados.0.smithi181.stdout:44: write initial oid 44 2024-04-26T01:24:05.568 INFO:tasks.rados.rados.0.smithi181.stdout:44: seq_num 43 ranges {0=737280} 2024-04-26T01:24:05.576 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:05.576 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:05.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:05.576 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s3 35/30 ECSubWrite(tid=32, reqid=client.4589.0:62, at_version=0'0, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.374598+0000", 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.20670371300000001, 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0017451470000000001, 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374598+0000", 2024-04-26T01:24:05.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374598+0000", 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374599+0000", 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1519999999999999e-06 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374614+0000", 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4781e-05 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374631+0000", 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7538e-05 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374643+0000", 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1294999999999999e-05 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374749+0000", 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000106189 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374762+0000", 2024-04-26T01:24:05.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3484e-05 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.376313+0000", 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001550266 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.376343+0000", 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0442000000000001e-05 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.581 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.7s3 35/30 ECSubRead(tid=33, to_read={3:e977e128:::smithi18128752-21:head=0,4096,0}, subchunks={3:e977e128:::smithi18128752-21:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.374673+0000", 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.20662878100000001, 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001635807, 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374673+0000", 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374673+0000", 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374673+0000", 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9200000000000002e-07 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374682+0000", 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7819999999999999e-06 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374694+0000", 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2164000000000001e-05 2024-04-26T01:24:05.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.374704+0000", 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0006999999999999e-05 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.375001+0000", 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029630699999999999 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:05.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.376309+0000", 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013081550000000001 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 35/30 ECSubWrite(tid=9, reqid=client.4589.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.429603+0000", 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.15169864, 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012696669999999999, 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.586 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429603+0000", 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429603+0000", 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429605+0000", 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8330000000000001e-06 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429627+0000", 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2286999999999999e-05 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429653+0000", 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5630999999999999e-05 2024-04-26T01:24:05.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429666+0000", 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.36e-05 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429760+0000", 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3374000000000004e-05 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.429768+0000", 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1880000000000002e-06 2024-04-26T01:24:05.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430797+0000", 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001029112 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430873+0000", 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5642000000000005e-05 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.8s1 35/30 ECSubRead(tid=10, to_read={3:1945891d:::smithi18128752-34:head=0,4096,0}, subchunks={3:1945891d:::smithi18128752-34:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.430940+0000", 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.15036126699999999, 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00039411600000000002, 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430940+0000", 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430940+0000", 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430941+0000", 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5499999999999997e-07 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430952+0000", 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1148000000000001e-05 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430970+0000", 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7822e-05 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.430983+0000", 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2680999999999999e-05 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.431076+0000", 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3097000000000007e-05 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.431335+0000", 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025851299999999998 2024-04-26T01:24:05.594 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 35/34 ECSubWrite(tid=6, reqid=client.4589.0:70, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.440881+0000", 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.14042105299999999, 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0019619540000000001, 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.440881+0000", 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.440881+0000", 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.440882+0000", 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0699999999999996e-07 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.441051+0000", 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016932899999999999 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.441066+0000", 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5563e-05 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.441075+0000", 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6500000000000002e-06 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.441202+0000", 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012663299999999999 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.441226+0000", 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.41e-05 2024-04-26T01:24:05.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.442770+0000", 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001544639 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.442843+0000", 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2132999999999995e-05 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.599 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.600 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.600 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:05.600 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:05.601 INFO:tasks.rados.rados.0.smithi181.stdout:44: writing smithi18128752-44 from 0 to 737280 tid 1 2024-04-26T01:24:05.601 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.601 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:05.602 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:24:05.611 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:05.612 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:05.612 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:05.616+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:05.636 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:05.637 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:05.654 INFO:tasks.rados.rados.0.smithi181.stdout:39: finishing write tid 1 to smithi18128752-39 2024-04-26T01:24:05.662 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:05.663 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:37: finishing write tid 3 to smithi18128752-37 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:37: oid 37 updating version 0 to 3 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:37: oid 37 updating version 3 to 4 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:37: done (15 left) 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:45: write initial oid 45 2024-04-26T01:24:05.671 INFO:tasks.rados.rados.0.smithi181.stdout:45: seq_num 44 ranges {0=753664} 2024-04-26T01:24:05.683 INFO:tasks.rados.rados.0.smithi181.stdout:45: writing smithi18128752-45 from 0 to 753664 tid 1 2024-04-26T01:24:05.684 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.684 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:05.684 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:05.706 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:05.708+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:05.709 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:05.709 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:05.734 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:121 3.ds0 3:b6ed1d05:::smithi18128752-41:head [append 0~688128 in=688128b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.492364+0000", 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.24733904300000001, 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.24737483599999999, 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 121 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.492364+0000", 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.492364+0000", 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.492366+0000", 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1629999999999999e-06 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493473+0000", 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001106903 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493493+0000", 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0482000000000001e-05 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493503+0000", 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6169999999999996e-06 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.593713+0000", 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10020987200000001 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.593955+0000", 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000242742 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595358+0000", 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014027690000000001 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697729+0000", 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10237120199999999 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.740 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:122 3.ds0 3:b6ed1d05:::smithi18128752-41:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.493522+0000", 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.24618019499999999, 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.246291384, 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 122 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493522+0000", 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493522+0000", 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493523+0000", 2024-04-26T01:24:05.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2399999999999998e-07 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493526+0000", 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3359999999999999e-06 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493532+0000", 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6239999999999999e-06 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493536+0000", 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3760000000000001e-06 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.696585+0000", 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20304873200000001 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.696668+0000", 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3269000000000005e-05 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697063+0000", 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00039460100000000001 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697824+0000", 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00076119200000000005 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.745 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:123 3.ds0 3:b6ed1d05:::smithi18128752-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.24615847099999999, 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.246333093, 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "waiting for rw locks", 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:05.746 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 123 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493545+0000", 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5299999999999999e-07 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493546+0000", 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7039999999999999e-06 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493552+0000", 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5119999999999999e-06 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493556+0000", 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2679999999999996e-06 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697434+0000", 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20387831000000001 2024-04-26T01:24:05.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697559+0000", 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012467700000000001 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 3 2024-04-26T01:24:05.750 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:05.751 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:24:05.754 INFO:tasks.rados.rados.0.smithi181.stdout:39: finishing write tid 2 to smithi18128752-39 2024-04-26T01:24:05.758 INFO:tasks.rados.rados.0.smithi181.stdout:35: finishing write tid 3 to smithi18128752-35 2024-04-26T01:24:05.758 INFO:tasks.rados.rados.0.smithi181.stdout:35: oid 35 updating version 0 to 4 2024-04-26T01:24:05.758 INFO:tasks.rados.rados.0.smithi181.stdout:35: oid 35 version 4 is already newer than 3 2024-04-26T01:24:05.758 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 4 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-26T01:24:05.758 INFO:tasks.rados.rados.0.smithi181.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-04-26T01:24:05.759 INFO:tasks.rados.rados.0.smithi181.stdout:35: done (15 left) 2024-04-26T01:24:05.759 INFO:tasks.rados.rados.0.smithi181.stdout:46: write initial oid 46 2024-04-26T01:24:05.759 INFO:tasks.rados.rados.0.smithi181.stdout:46: seq_num 45 ranges {0=770048} 2024-04-26T01:24:05.774 INFO:tasks.rados.rados.0.smithi181.stdout:46: writing smithi18128752-46 from 0 to 770048 tid 1 2024-04-26T01:24:05.775 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:05.775 INFO:tasks.rados.rados.0.smithi181.stdout:22: finishing write tid 1 to smithi18128752-22 2024-04-26T01:24:05.775 INFO:tasks.rados.rados.0.smithi181.stdout:41: finishing write tid 1 to smithi18128752-41 2024-04-26T01:24:05.778 INFO:tasks.rados.rados.0.smithi181.stdout:38: finishing write tid 1 to smithi18128752-38 2024-04-26T01:24:05.779 INFO:tasks.rados.rados.0.smithi181.stdout:38: finishing write tid 2 to smithi18128752-38 2024-04-26T01:24:05.780 INFO:tasks.rados.rados.0.smithi181.stdout:20: finishing write tid 1 to smithi18128752-20 2024-04-26T01:24:05.807 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:05.806+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:05.846 INFO:tasks.rados.rados.0.smithi181.stdout:24: finishing write tid 1 to smithi18128752-24 2024-04-26T01:24:05.878 INFO:tasks.rados.rados.0.smithi181.stdout:43: finishing write tid 1 to smithi18128752-43 2024-04-26T01:24:05.880 INFO:tasks.rados.rados.0.smithi181.stdout:20: finishing write tid 2 to smithi18128752-20 2024-04-26T01:24:05.907 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:05.908+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:05.938 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:05.938 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:05.938 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:05.938 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:05.938 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:05.947 INFO:tasks.rados.rados.0.smithi181.stdout:24: finishing write tid 2 to smithi18128752-24 2024-04-26T01:24:05.952 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:24:05.963 INFO:tasks.rados.rados.0.smithi181.stdout:44: finishing write tid 1 to smithi18128752-44 2024-04-26T01:24:05.963 INFO:tasks.rados.rados.0.smithi181.stdout:44: finishing write tid 2 to smithi18128752-44 2024-04-26T01:24:05.972 INFO:tasks.rados.rados.0.smithi181.stdout:43: finishing write tid 2 to smithi18128752-43 2024-04-26T01:24:05.982 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:05.982 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:06.008 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:06.008+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:06.009 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:06.009 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:06.036 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:24:06.047 INFO:tasks.rados.rados.0.smithi181.stdout:25: finishing write tid 1 to smithi18128752-25 2024-04-26T01:24:06.048 INFO:tasks.rados.rados.0.smithi181.stdout:25: finishing write tid 2 to smithi18128752-25 2024-04-26T01:24:06.066 INFO:tasks.rados.rados.0.smithi181.stdout:22: finishing write tid 2 to smithi18128752-22 2024-04-26T01:24:06.071 INFO:tasks.rados.rados.0.smithi181.stdout:40: finishing write tid 1 to smithi18128752-40 2024-04-26T01:24:06.082 INFO:tasks.rados.rados.0.smithi181.stdout:41: finishing write tid 2 to smithi18128752-41 2024-04-26T01:24:06.083 INFO:tasks.rados.rados.0.smithi181.stdout:46: finishing write tid 1 to smithi18128752-46 2024-04-26T01:24:06.084 INFO:tasks.rados.rados.0.smithi181.stdout:38: finishing write tid 3 to smithi18128752-38 2024-04-26T01:24:06.084 INFO:tasks.rados.rados.0.smithi181.stdout:38: oid 38 updating version 0 to 2 2024-04-26T01:24:06.085 INFO:tasks.rados.rados.0.smithi181.stdout:38: oid 38 version 2 is already newer than 1 2024-04-26T01:24:06.085 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 2 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-04-26T01:24:06.085 INFO:tasks.rados.rados.0.smithi181.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-04-26T01:24:06.085 INFO:tasks.rados.rados.0.smithi181.stdout:38: done (15 left) 2024-04-26T01:24:06.085 INFO:tasks.rados.rados.0.smithi181.stdout:47: write initial oid 47 2024-04-26T01:24:06.085 INFO:tasks.rados.rados.0.smithi181.stdout:47: seq_num 46 ranges {0=786432} 2024-04-26T01:24:06.087 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:06.087 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:06.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:06.088 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:06.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.088 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 35/30 ECSubWriteReply(tid=34, last_complete=35'4, committed=1, applied=1))", 2024-04-26T01:24:06.088 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.475085+0000", 2024-04-26T01:24:06.088 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.612249551, 2024-04-26T01:24:06.088 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099997831999999995, 2024-04-26T01:24:06.089 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.089 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.089 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.090 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.475085+0000", 2024-04-26T01:24:06.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.475085+0000", 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.475086+0000", 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5799999999999998e-07 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.475098+0000", 2024-04-26T01:24:06.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2459999999999999e-05 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.475114+0000", 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6405e-05 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.475129+0000", 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4997000000000001e-05 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575028+0000", 2024-04-26T01:24:06.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099898985999999995 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575082+0000", 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4026000000000001e-05 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/34 ECSubWrite(tid=165, reqid=client.4589.0:113, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.479948+0000", 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.60738609899999996, 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10254308199999999, 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.479948+0000", 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.479948+0000", 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.479948+0000", 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.34e-07 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.479965+0000", 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6966999999999999e-05 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.479975+0000", 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5889999999999992e-06 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.479982+0000", 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2599999999999999e-06 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.480856+0000", 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00087418800000000003 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.480868+0000", 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1413999999999999e-05 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.582434+0000", 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101566396 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.582491+0000", 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6934000000000001e-05 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.098 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.bs0 35/34 ECSubReadReply(tid=35, attrs_read=0))", 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.480920+0000", 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.60641371399999999, 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100686095, 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.480920+0000", 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.480920+0000", 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.480921+0000", 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5499999999999996e-07 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.480983+0000", 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1669999999999997e-05 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.480999+0000", 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5464e-05 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481007+0000", 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4719999999999995e-06 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.581576+0000", 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100569302 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.581607+0000", 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0232000000000001e-05 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.bs0 35/34 ECSubReadReply(tid=35, attrs_read=0))", 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.481102+0000", 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.606231941, 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10139448399999999, 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.102 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481102+0000", 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481102+0000", 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481103+0000", 2024-04-26T01:24:06.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0699999999999996e-07 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481371+0000", 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026784500000000001 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481384+0000", 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.31e-05 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481392+0000", 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3729999999999993e-06 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.581721+0000", 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10032845999999999 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.582497+0000", 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00077579899999999998 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 35/30 ECSubWriteReply(tid=34, last_complete=35'4, committed=1, applied=1))", 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.481815+0000", 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.60551880599999997, 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.193467308, 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481815+0000", 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481815+0000", 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481816+0000", 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1900000000000002e-07 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481821+0000", 2024-04-26T01:24:06.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4630000000000003e-06 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481831+0000", 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0006999999999999e-05 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.481837+0000", 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3670000000000002e-06 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675259+0000", 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19342221900000001 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675283+0000", 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3533e-05 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:121 3.ds0 3:b6ed1d05:::smithi18128752-41:head [append 0~688128 in=688128b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.492364+0000", 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.59497060400000001, 2024-04-26T01:24:06.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.27543901700000001, 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 121 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.492364+0000", 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.492364+0000", 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.492366+0000", 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1629999999999999e-06 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493473+0000", 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001106903 2024-04-26T01:24:06.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493493+0000", 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0482000000000001e-05 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493503+0000", 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6169999999999996e-06 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.593713+0000", 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10020987200000001 2024-04-26T01:24:06.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.593955+0000", 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000242742 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595358+0000", 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014027690000000001 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697729+0000", 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10237120199999999 2024-04-26T01:24:06.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.767731+0000", 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.070002062000000004 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.767803+0000", 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1204999999999993e-05 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.1s0 35/30 ECSubRead(tid=36, to_read={3:865c3313:::smithi18128752-37:head=0,4096,0}, subchunks={3:865c3313:::smithi18128752-37:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.574727+0000", 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.51260682000000002, 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100887158, 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.574727+0000", 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.574727+0000", 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.574727+0000", 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.574727+0000", 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.574759+0000", 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1779999999999997e-05 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.574770+0000", 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0747999999999999e-05 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675323+0000", 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10055275599999999 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675614+0000", 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029187400000000002 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 35/30 ECSubWriteReply(tid=34, last_complete=35'4, committed=1, applied=1))", 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.575501+0000", 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.51183331200000004, 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100159421, 2024-04-26T01:24:06.118 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575501+0000", 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575501+0000", 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.119 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575502+0000", 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0100000000000001e-06 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575511+0000", 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8079999999999994e-06 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575525+0000", 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4739000000000001e-05 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575534+0000", 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8610000000000004e-06 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675643+0000", 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100108661 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675660+0000", 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7342000000000001e-05 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.121 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.1s0 35/30 ECSubReadReply(tid=36, attrs_read=0))", 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.575519+0000", 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.51181531800000002, 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100349359, 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575519+0000", 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575519+0000", 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575520+0000", 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.063e-06 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575531+0000", 2024-04-26T01:24:06.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1209999999999999e-05 2024-04-26T01:24:06.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.124 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.124 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.124 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575541+0000", 2024-04-26T01:24:06.124 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0346999999999999e-05 2024-04-26T01:24:06.124 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575548+0000", 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2020000000000001e-06 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675835+0000", 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10028693800000001 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.675868+0000", 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3599000000000001e-05 2024-04-26T01:24:06.125 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.1s0 35/30 ECSubReadReply(tid=36, attrs_read=0))", 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.575622+0000", 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.51171207600000002, 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10060777999999999, 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.126 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575622+0000", 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575622+0000", 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575623+0000", 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0999999999999998e-07 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.127 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575631+0000", 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8140000000000001e-06 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575641+0000", 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9129999999999998e-06 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.575646+0000", 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8069999999999996e-06 2024-04-26T01:24:06.128 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.676004+0000", 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10035809499999999 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.676230+0000", 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000225441 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.129 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/30 ECSubWriteReply(tid=37, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.599376+0000", 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.48795859000000003, 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098277762000000005, 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599376+0000", 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599376+0000", 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599377+0000", 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.057e-06 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599382+0000", 2024-04-26T01:24:06.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5990000000000001e-06 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599398+0000", 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6067999999999999e-05 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599410+0000", 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1377e-05 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697629+0000", 2024-04-26T01:24:06.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098219614999999996 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697653+0000", 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4046e-05 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/30 ECSubWriteReply(tid=37, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.599472+0000", 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.487861763, 2024-04-26T01:24:06.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098258769999999995, 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599472+0000", 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599472+0000", 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599473+0000", 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.079e-06 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599477+0000", 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5729999999999999e-06 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599488+0000", 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1070000000000001e-05 2024-04-26T01:24:06.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599494+0000", 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8189999999999997e-06 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697704+0000", 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098210224999999998 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697731+0000", 2024-04-26T01:24:06.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7004000000000001e-05 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/30 ECSubWriteReply(tid=37, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.600748+0000", 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.48658620299999999, 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.097026954999999998, 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.600748+0000", 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.600748+0000", 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.600749+0000", 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.37e-07 2024-04-26T01:24:06.138 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.600752+0000", 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0790000000000001e-06 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.600762+0000", 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8490000000000001e-06 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.600767+0000", 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0839999999999999e-06 2024-04-26T01:24:06.139 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697760+0000", 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096993432000000004 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697775+0000", 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4874e-05 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.140 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/30 ECSubWriteReply(tid=37, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.601059+0000", 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.48627515700000001, 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096762915000000005, 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.601059+0000", 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.141 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.601059+0000", 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.601060+0000", 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1699999999999998e-07 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.601063+0000", 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1190000000000001e-06 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.142 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.601071+0000", 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.601075+0000", 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2690000000000001e-06 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697801+0000", 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096726257999999996 2024-04-26T01:24:06.143 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697822+0000", 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0707e-05 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 35/30 ECSubWriteReply(tid=39, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.784224+0000", 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.30311059200000001, 2024-04-26T01:24:06.144 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.095364237000000004, 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784224+0000", 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784224+0000", 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.145 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784225+0000", 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5099999999999998e-07 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784231+0000", 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6089999999999997e-06 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784243+0000", 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1552000000000001e-05 2024-04-26T01:24:06.146 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784251+0000", 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8539999999999997e-06 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.879568+0000", 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.095317137999999996 2024-04-26T01:24:06.147 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.879588+0000", 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0132999999999998e-05 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 35/30 ECSubWriteReply(tid=39, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.784311+0000", 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.30302319, 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10025391, 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.148 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784311+0000", 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784311+0000", 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.149 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784312+0000", 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0440000000000001e-06 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784318+0000", 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4640000000000003e-06 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784328+0000", 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0780000000000002e-06 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.150 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.884429+0000", 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100101868 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.884554+0000", 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012443799999999999 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.884565+0000", 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1018e-05 2024-04-26T01:24:06.151 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 35/30 ECSubWriteReply(tid=39, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.784823+0000", 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.30251114800000001, 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.094799094, 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.152 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784823+0000", 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784823+0000", 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784824+0000", 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7899999999999997e-07 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.153 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784830+0000", 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1229999999999997e-06 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784847+0000", 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6608000000000001e-05 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784870+0000", 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2940000000000001e-05 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.154 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.879612+0000", 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.094742380000000001 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.879622+0000", 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0164e-05 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.155 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/34 ECSubWrite(tid=168, reqid=client.4589.0:113, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.784926+0000", 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.30240803300000002, 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10177354199999999, 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784926+0000", 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.156 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784926+0000", 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784927+0000", 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3600000000000003e-07 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784942+0000", 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5062000000000001e-05 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.157 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784963+0000", 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0941000000000002e-05 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784975+0000", 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2339e-05 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785138+0000", 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016279700000000001 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.158 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785163+0000", 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5426999999999999e-05 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.886662+0000", 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10149819 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.886700+0000", 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8050000000000003e-05 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.159 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 35/30 ECSubWriteReply(tid=40, last_complete=35'6, committed=1, applied=1))", 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.785810+0000", 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.301524546, 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098719471000000003, 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785810+0000", 2024-04-26T01:24:06.160 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785810+0000", 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785810+0000", 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1299999999999999e-07 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785815+0000", 2024-04-26T01:24:06.161 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7269999999999998e-06 2024-04-26T01:24:06.162 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.162 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.163 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.163 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785829+0000", 2024-04-26T01:24:06.163 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3492000000000001e-05 2024-04-26T01:24:06.164 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.164 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.164 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.164 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.785837+0000", 2024-04-26T01:24:06.164 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8089999999999999e-06 2024-04-26T01:24:06.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.165 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.165 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.165 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.884511+0000", 2024-04-26T01:24:06.165 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098673538000000005 2024-04-26T01:24:06.165 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.166 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.166 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.166 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.884529+0000", 2024-04-26T01:24:06.166 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8091999999999999e-05 2024-04-26T01:24:06.166 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.166 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.167 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.167 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.167 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.167 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.3s2 35/34 ECSubRead(tid=169, to_read={3:c5d16e9b:::smithi18128752-38:head=0,4096,0}, subchunks={3:c5d16e9b:::smithi18128752-38:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:06.167 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.786022+0000", 2024-04-26T01:24:06.167 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.30131227700000002, 2024-04-26T01:24:06.168 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10055462499999999, 2024-04-26T01:24:06.168 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.168 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.168 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.168 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.168 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.169 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.786022+0000", 2024-04-26T01:24:06.169 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.169 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.169 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.169 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.170 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.786022+0000", 2024-04-26T01:24:06.170 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.170 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.170 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.170 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.171 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.786023+0000", 2024-04-26T01:24:06.171 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0900000000000001e-07 2024-04-26T01:24:06.171 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.171 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.171 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.171 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.786032+0000", 2024-04-26T01:24:06.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0690000000000008e-06 2024-04-26T01:24:06.172 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.172 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.172 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.172 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.786043+0000", 2024-04-26T01:24:06.172 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1168e-05 2024-04-26T01:24:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.886125+0000", 2024-04-26T01:24:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100081789 2024-04-26T01:24:06.173 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.886348+0000", 2024-04-26T01:24:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022336799999999999 2024-04-26T01:24:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.174 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.175 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.175 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.886577+0000", 2024-04-26T01:24:06.175 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022852199999999999 2024-04-26T01:24:06.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.175 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.175 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.176 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.176 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.176 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:06.181 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:24:06.181 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:06.180+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:06.185 INFO:tasks.rados.rados.0.smithi181.stdout:47: writing smithi18128752-47 from 0 to 786432 tid 1 2024-04-26T01:24:06.185 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:06.185 INFO:tasks.rados.rados.0.smithi181.stdout:20: finishing write tid 3 to smithi18128752-20 2024-04-26T01:24:06.185 INFO:tasks.rados.rados.0.smithi181.stdout:20: oid 20 updating version 0 to 1 2024-04-26T01:24:06.186 INFO:tasks.rados.rados.0.smithi181.stdout:20: oid 20 updating version 1 to 2 2024-04-26T01:24:06.186 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 2 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-04-26T01:24:06.186 INFO:tasks.rados.rados.0.smithi181.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-04-26T01:24:06.186 INFO:tasks.rados.rados.0.smithi181.stdout:20: done (15 left) 2024-04-26T01:24:06.186 INFO:tasks.rados.rados.0.smithi181.stdout:48: write initial oid 48 2024-04-26T01:24:06.186 INFO:tasks.rados.rados.0.smithi181.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-04-26T01:24:06.187 INFO:tasks.rados.rados.0.smithi181.stdout:48: writing smithi18128752-48 from 0 to 425984 tid 1 2024-04-26T01:24:06.187 INFO:tasks.rados.rados.0.smithi181.stdout:48: writing smithi18128752-48 from 425984 to 802816 tid 2 2024-04-26T01:24:06.187 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:06.187 INFO:tasks.rados.rados.0.smithi181.stdout:24: finishing write tid 3 to smithi18128752-24 2024-04-26T01:24:06.187 INFO:tasks.rados.rados.0.smithi181.stdout:24: oid 24 updating version 0 to 2 2024-04-26T01:24:06.187 INFO:tasks.rados.rados.0.smithi181.stdout:24: oid 24 version 2 is already newer than 1 2024-04-26T01:24:06.188 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 2 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-04-26T01:24:06.188 INFO:tasks.rados.rados.0.smithi181.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:24:06.188 INFO:tasks.rados.rados.0.smithi181.stdout:24: done (15 left) 2024-04-26T01:24:06.188 INFO:tasks.rados.rados.0.smithi181.stdout:49: write initial oid 49 2024-04-26T01:24:06.188 INFO:tasks.rados.rados.0.smithi181.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-04-26T01:24:06.188 INFO:tasks.rados.rados.0.smithi181.stdout:49: writing smithi18128752-49 from 0 to 442368 tid 1 2024-04-26T01:24:06.189 INFO:tasks.rados.rados.0.smithi181.stdout:49: writing smithi18128752-49 from 442368 to 819200 tid 2 2024-04-26T01:24:06.189 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:06.189 INFO:tasks.rados.rados.0.smithi181.stdout:39: finishing write tid 3 to smithi18128752-39 2024-04-26T01:24:06.189 INFO:tasks.rados.rados.0.smithi181.stdout:39: oid 39 updating version 0 to 3 2024-04-26T01:24:06.189 INFO:tasks.rados.rados.0.smithi181.stdout:39: oid 39 updating version 3 to 4 2024-04-26T01:24:06.189 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 39 v 4 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-04-26T01:24:06.190 INFO:tasks.rados.rados.0.smithi181.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-04-26T01:24:06.190 INFO:tasks.rados.rados.0.smithi181.stdout:39: done (15 left) 2024-04-26T01:24:06.190 INFO:tasks.rados.rados.0.smithi181.stdout:50: write initial oid 50 2024-04-26T01:24:06.190 INFO:tasks.rados.rados.0.smithi181.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-04-26T01:24:06.190 INFO:tasks.rados.rados.0.smithi181.stdout:50: writing smithi18128752-50 from 0 to 458752 tid 1 2024-04-26T01:24:06.191 INFO:tasks.rados.rados.0.smithi181.stdout:50: writing smithi18128752-50 from 458752 to 835584 tid 2 2024-04-26T01:24:06.191 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:06.191 INFO:tasks.rados.rados.0.smithi181.stdout:40: finishing write tid 2 to smithi18128752-40 2024-04-26T01:24:06.191 INFO:tasks.rados.rados.0.smithi181.stdout:46: finishing write tid 2 to smithi18128752-46 2024-04-26T01:24:06.267 INFO:tasks.rados.rados.0.smithi181.stdout:44: finishing write tid 3 to smithi18128752-44 2024-04-26T01:24:06.267 INFO:tasks.rados.rados.0.smithi181.stdout:44: oid 44 updating version 0 to 5 2024-04-26T01:24:06.267 INFO:tasks.rados.rados.0.smithi181.stdout:44: oid 44 updating version 5 to 6 2024-04-26T01:24:06.268 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-04-26T01:24:06.268 INFO:tasks.rados.rados.0.smithi181.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-04-26T01:24:06.268 INFO:tasks.rados.rados.0.smithi181.stdout:44: done (15 left) 2024-04-26T01:24:06.268 INFO:tasks.rados.rados.0.smithi181.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2024-04-26T01:24:06.268 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:06.278 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:06.278+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:06.344 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=16, reqid=client.4589.0:86, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.16249740400000001, 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.16254058099999999, 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187013+0000", 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8840000000000001e-06 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187073+0000", 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9972000000000001e-05 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187121+0000", 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7222999999999998e-05 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187151+0000", 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0107000000000002e-05 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284476+0000", 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.097325196000000003 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284487+0000", 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1633e-05 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=17, reqid=client.4589.0:146, at_version=35'9, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.15913002000000001, 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.159278636, 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190380+0000", 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5099999999999999e-06 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190583+0000", 2024-04-26T01:24:06.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020251199999999999 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190609+0000", 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6256999999999999e-05 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.351 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190624+0000", 2024-04-26T01:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5302999999999999e-05 2024-04-26T01:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.352 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.156325462, 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.15655190099999999, 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.353 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.354 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193185+0000", 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193366+0000", 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000181096 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.355 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193393+0000", 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6673e-05 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193408+0000", 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.554e-05 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=16, reqid=client.4589.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.356 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.093549112000000004, 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.093850249999999996, 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.357 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1800000000000001e-07 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255984+0000", 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4032e-05 2024-04-26T01:24:06.358 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255996+0000", 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1933000000000001e-05 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.256005+0000", 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1449999999999997e-06 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.359 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.360 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T01:24:06.360 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 4 2024-04-26T01:24:06.360 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:06.361 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:24:06.368 INFO:tasks.rados.rados.0.smithi181.stdout:42: finishing write tid 1 to smithi18128752-42 2024-04-26T01:24:06.368 INFO:tasks.rados.rados.0.smithi181.stdout:42: finishing write tid 2 to smithi18128752-42 2024-04-26T01:24:06.369 INFO:tasks.rados.rados.0.smithi181.stdout:45: finishing write tid 1 to smithi18128752-45 2024-04-26T01:24:06.374 INFO:tasks.rados.rados.0.smithi181.stdout:45: finishing write tid 2 to smithi18128752-45 2024-04-26T01:24:06.379 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:06.380+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:06.480 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:06.484+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:06.483 INFO:tasks.rados.rados.0.smithi181.stdout:43: finishing write tid 3 to smithi18128752-43 2024-04-26T01:24:06.483 INFO:tasks.rados.rados.0.smithi181.stdout:43: oid 43 updating version 0 to 8 2024-04-26T01:24:06.483 INFO:tasks.rados.rados.0.smithi181.stdout:43: oid 43 version 8 is already newer than 7 2024-04-26T01:24:06.483 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 8 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-26T01:24:06.483 INFO:tasks.rados.rados.0.smithi181.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-04-26T01:24:06.484 INFO:tasks.rados.rados.0.smithi181.stdout:43: done (15 left) 2024-04-26T01:24:06.484 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 3 current snap is 0 2024-04-26T01:24:06.484 INFO:tasks.rados.rados.0.smithi181.stdout:52: seq_num 50 ranges {49152=475136,524288=376832} 2024-04-26T01:24:06.491 INFO:tasks.rados.rados.0.smithi181.stdout:52: writing smithi18128752-3 from 49152 to 524288 tid 1 2024-04-26T01:24:06.494 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:06.494 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:06.494 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:06.494 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:06.495 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:06.498 INFO:tasks.rados.rados.0.smithi181.stdout:52: writing smithi18128752-3 from 524288 to 901120 tid 2 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:46: finishing write tid 3 to smithi18128752-46 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:46: oid 46 updating version 0 to 6 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:46: oid 46 version 6 is already newer than 5 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:46: done (15 left) 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:53: rmattr oid 35 current snap is 0 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:53: done (15 left) 2024-04-26T01:24:06.499 INFO:tasks.rados.rados.0.smithi181.stdout:54: delete oid 34 current snap is 0 2024-04-26T01:24:06.509 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:24:06.581 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:06.580+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:06.639 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 35/30 ECSubWrite(tid=37, reqid=client.4589.0:121, at_version=35'5, trim_to=0'0, roll_forward_to=34'4))", 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.595627+0000", 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.0457567569999999, 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003799349, 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595627+0000", 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595627+0000", 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595629+0000", 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9810000000000002e-06 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595756+0000", 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000127031 2024-04-26T01:24:06.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595792+0000", 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6130000000000001e-05 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595809+0000", 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7122999999999999e-05 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595951+0000", 2024-04-26T01:24:06.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014112599999999999 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.595965+0000", 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4241e-05 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599390+0000", 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0034254670000000002 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.599427+0000", 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.625e-05 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=9, reqid=client.4589.0:71, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.663974+0000", 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.97740996300000005, 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001005419, 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.663974+0000", 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.663974+0000", 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.663974+0000", 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6800000000000001e-07 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.663994+0000", 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.916e-05 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.664002+0000", 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1769999999999998e-06 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.664010+0000", 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.439e-06 2024-04-26T01:24:06.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.664142+0000", 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013187100000000001 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.664157+0000", 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4955e-05 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.664925+0000", 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00076766400000000004 2024-04-26T01:24:06.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.664979+0000", 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4784999999999998e-05 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 35/30 ECSubWrite(tid=38, reqid=client.4589.0:122, at_version=35'6, trim_to=0'0, roll_forward_to=34'4))", 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.697095+0000", 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.94428850499999994, 2024-04-26T01:24:06.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00076784499999999996, 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697095+0000", 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697095+0000", 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697097+0000", 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.906e-06 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697120+0000", 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2892e-05 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697140+0000", 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9570000000000001e-05 2024-04-26T01:24:06.650 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697151+0000", 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0747999999999999e-05 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697254+0000", 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010388800000000001 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697262+0000", 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6190000000000002e-06 2024-04-26T01:24:06.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697835+0000", 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00057341100000000002 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697863+0000", 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7810999999999999e-05 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.652 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=10, reqid=client.4589.0:73, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.747965+0000", 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.89341928100000001, 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.11560028899999999, 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.747965+0000", 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.653 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.747965+0000", 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.747967+0000", 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4310000000000001e-06 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.748283+0000", 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000316304 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.654 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.748320+0000", 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7088000000000001e-05 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848434+0000", 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100113408 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848558+0000", 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012380299999999999 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848568+0000", 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9729999999999992e-06 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.863533+0000", 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014965216999999999 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.863565+0000", 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2064999999999999e-05 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.656 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 35/30 ECSubWriteReply(tid=10, last_complete=34'6, committed=1, applied=1))", 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.763240+0000", 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.87814364300000003, 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100287714, 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.763240+0000", 2024-04-26T01:24:06.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.763240+0000", 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.763242+0000", 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9700000000000002e-06 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.763256+0000", 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3594000000000001e-05 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.763284+0000", 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7792000000000001e-05 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.763314+0000", 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9867000000000001e-05 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.863490+0000", 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10017617299999999 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.863528+0000", 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8318000000000001e-05 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=9, reqid=client.4589.0:64, at_version=0'0, trim_to=0'0, roll_forward_to=35'3))", 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.766109+0000", 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.87527517499999996, 2024-04-26T01:24:06.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001047409, 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766109+0000", 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766109+0000", 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766110+0000", 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.66e-06 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766145+0000", 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4459000000000003e-05 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766169+0000", 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4584e-05 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766186+0000", 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6860000000000001e-05 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766250+0000", 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3705000000000001e-05 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.766258+0000", 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.605e-06 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.767091+0000", 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00083340400000000005 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.767156+0000", 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5131999999999999e-05 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.664 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=11, reqid=client.4589.0:74, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.848489+0000", 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.79289536299999996, 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.015095928999999999, 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848489+0000", 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848489+0000", 2024-04-26T01:24:06.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848489+0000", 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5899999999999996e-07 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848518+0000", 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8343999999999998e-05 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848532+0000", 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3973e-05 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.848541+0000", 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5170000000000005e-06 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.849146+0000", 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00060500400000000002 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.849153+0000", 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0480000000000001e-06 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.863569+0000", 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.014416112999999999 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.863585+0000", 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5271e-05 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=10, reqid=client.4589.0:65, at_version=35'4, trim_to=0'0, roll_forward_to=35'3))", 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.868575+0000", 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.77280903300000003, 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00077956399999999995, 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.669 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868575+0000", 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868575+0000", 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868576+0000", 2024-04-26T01:24:06.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.356e-06 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868608+0000", 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1854999999999999e-05 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868626+0000", 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8125999999999999e-05 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868635+0000", 2024-04-26T01:24:06.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9369999999999993e-06 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868718+0000", 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2956999999999999e-05 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.868726+0000", 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8930000000000005e-06 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.869319+0000", 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059346300000000002 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.869354+0000", 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4977000000000001e-05 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.cs1 35/34 ECSubRead(tid=12, to_read={3:3c2c778f:::smithi18128752-24:head=0,4096,0}, subchunks={3:3c2c778f:::smithi18128752-24:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.953258+0000", 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.68812556300000005, 2024-04-26T01:24:06.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100625069, 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953258+0000", 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953258+0000", 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953261+0000", 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2670000000000001e-06 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953281+0000", 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0602999999999999e-05 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953321+0000", 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9922999999999997e-05 2024-04-26T01:24:06.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953346+0000", 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5032999999999999e-05 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.053577+0000", 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10023045799999999 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.053883+0000", 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030678500000000001 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.676 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=13, reqid=client.4589.0:74, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.58719191999999998, 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20101313800000001, 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054193+0000", 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.13e-06 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054208+0000", 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4562000000000001e-05 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054223+0000", 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5536000000000002e-05 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.154317+0000", 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100094023 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.254494+0000", 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100176788 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.254508+0000", 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3779e-05 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255170+0000", 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00066210200000000002 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255205+0000", 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5218000000000001e-05 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=11, reqid=client.4589.0:65, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.072898+0000", 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.56848625100000005, 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00088159499999999999, 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.072898+0000", 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.072898+0000", 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.072899+0000", 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.778e-06 2024-04-26T01:24:06.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.072925+0000", 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5248e-05 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.072942+0000", 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7090000000000001e-05 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.072956+0000", 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4371e-05 2024-04-26T01:24:06.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.073068+0000", 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011134200000000001 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.073078+0000", 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.004e-05 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.073735+0000", 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00065695599999999999 2024-04-26T01:24:06.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.073779+0000", 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.477e-05 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=12, reqid=client.4589.0:67, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.175981+0000", 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.46540282599999999, 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0019327770000000001, 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.685 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.175981+0000", 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.175981+0000", 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.175984+0000", 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.48e-06 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.176106+0000", 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012236300000000001 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.176137+0000", 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1202999999999999e-05 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.176155+0000", 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8003e-05 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.176342+0000", 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000186479 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.176360+0000", 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8810000000000001e-05 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.177839+0000", 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014789429999999999 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.177914+0000", 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.4496e-05 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=13, reqid=client.4589.0:68, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.178670+0000", 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.46271372, 2024-04-26T01:24:06.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001229434, 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178670+0000", 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178670+0000", 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178672+0000", 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4300000000000001e-06 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178719+0000", 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6859e-05 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178743+0000", 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4029999999999999e-05 2024-04-26T01:24:06.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178758+0000", 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5237e-05 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178896+0000", 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013793200000000001 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.178910+0000", 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4433000000000001e-05 2024-04-26T01:24:06.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.179831+0000", 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00092093399999999999 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.179900+0000", 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8578999999999995e-05 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.693 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=15, reqid=client.4589.0:85, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.184060+0000", 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.45732412300000003, 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0053253950000000001, 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.184060+0000", 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.184060+0000", 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.184061+0000", 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.584e-06 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.184233+0000", 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017188899999999999 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.184261+0000", 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7753e-05 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.184277+0000", 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6042999999999999e-05 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187313+0000", 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0030357940000000001 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187340+0000", 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7019e-05 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.189277+0000", 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0019369809999999999 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.189385+0000", 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010833200000000001 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.697 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=16, reqid=client.4589.0:86, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.45437251200000001, 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.1988666, 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187013+0000", 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8840000000000001e-06 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187073+0000", 2024-04-26T01:24:06.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9972000000000001e-05 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187121+0000", 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7222999999999998e-05 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187151+0000", 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0107000000000002e-05 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284476+0000", 2024-04-26T01:24:06.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.097325196000000003 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284487+0000", 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1633e-05 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385837+0000", 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10134913199999999 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385878+0000", 2024-04-26T01:24:06.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1452999999999998e-05 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=17, reqid=client.4589.0:146, at_version=35'9, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.45100512799999998, 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19552889900000001, 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190380+0000", 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5099999999999999e-06 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190583+0000", 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020251199999999999 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190609+0000", 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6256999999999999e-05 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190624+0000", 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5302999999999999e-05 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.384789+0000", 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.194164533 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.384800+0000", 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0919e-05 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385883+0000", 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001082902 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385908+0000", 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4963000000000001e-05 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.44820057000000002, 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19307022800000001, 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.706 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193185+0000", 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193366+0000", 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000181096 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193393+0000", 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6673e-05 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193408+0000", 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.554e-05 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385391+0000", 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19198310199999999 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385402+0000", 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0586e-05 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.386222+0000", 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081973500000000004 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.386254+0000", 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1866000000000003e-05 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=15, reqid=client.4589.0:76, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.255851+0000", 2024-04-26T01:24:06.710 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.38553314799999999, 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001193693, 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255851+0000", 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255851+0000", 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255851+0000", 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2800000000000002e-07 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255920+0000", 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8552000000000004e-05 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255933+0000", 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3135999999999999e-05 2024-04-26T01:24:06.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255943+0000", 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0158e-05 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.256213+0000", 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000270365 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.256220+0000", 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2960000000000004e-06 2024-04-26T01:24:06.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.257020+0000", 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00080069799999999997 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.257044+0000", 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.406e-05 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=16, reqid=client.4589.0:77, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.38542421999999998, 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10149490999999999, 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255960+0000", 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1800000000000001e-07 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255984+0000", 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4032e-05 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255996+0000", 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1933000000000001e-05 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.256005+0000", 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1449999999999997e-06 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.356307+0000", 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10030204700000001 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.356320+0000", 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2751e-05 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.357384+0000", 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010635040000000001 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.357455+0000", 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1080000000000004e-05 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.718 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 35/30 ECSubWrite(tid=46, reqid=client.4589.0:139, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.289998+0000", 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.35138609199999998, 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001619037, 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289998+0000", 2024-04-26T01:24:06.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289998+0000", 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289999+0000", 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.533e-06 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290238+0000", 2024-04-26T01:24:06.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000238693 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290266+0000", 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8314e-05 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290281+0000", 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4185000000000001e-05 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290416+0000", 2024-04-26T01:24:06.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000135048 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290446+0000", 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0012e-05 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291576+0000", 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001130668 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291617+0000", 2024-04-26T01:24:06.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0584000000000003e-05 2024-04-26T01:24:06.723 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.723 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.723 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.723 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:06.723 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:06.723 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:06.724 INFO:tasks.rados.rados.0.smithi181.stdout:40: finishing write tid 3 to smithi18128752-40 2024-04-26T01:24:06.724 INFO:tasks.rados.rados.0.smithi181.stdout:40: oid 40 updating version 0 to 11 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:40: oid 40 updating version 11 to 12 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:42: finishing write tid 3 to smithi18128752-42 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:42: oid 42 updating version 0 to 13 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:42: oid 42 updating version 13 to 14 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 42 v 14 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:51: finishing copy_from racing read to smithi18128752-2 2024-04-26T01:24:06.725 INFO:tasks.rados.rados.0.smithi181.stdout:47: finishing write tid 1 to smithi18128752-47 2024-04-26T01:24:06.726 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:24:06.726 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:06.724+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:06.757 INFO:tasks.rados.rados.0.smithi181.stdout:26: finishing write tid 1 to smithi18128752-26 2024-04-26T01:24:06.757 INFO:tasks.rados.rados.0.smithi181.stdout:26: finishing write tid 2 to smithi18128752-26 2024-04-26T01:24:06.771 INFO:tasks.rados.rados.0.smithi181.stdout:23: finishing write tid 1 to smithi18128752-23 2024-04-26T01:24:06.771 INFO:tasks.rados.rados.0.smithi181.stdout:23: finishing write tid 2 to smithi18128752-23 2024-04-26T01:24:06.771 INFO:tasks.rados.rados.0.smithi181.stdout:29: finishing write tid 1 to smithi18128752-29 2024-04-26T01:24:06.771 INFO:tasks.rados.rados.0.smithi181.stdout:45: finishing write tid 3 to smithi18128752-45 2024-04-26T01:24:06.772 INFO:tasks.rados.rados.0.smithi181.stdout:45: oid 45 updating version 0 to 16 2024-04-26T01:24:06.772 INFO:tasks.rados.rados.0.smithi181.stdout:45: oid 45 version 16 is already newer than 15 2024-04-26T01:24:06.772 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 16 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-04-26T01:24:06.772 INFO:tasks.rados.rados.0.smithi181.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-04-26T01:24:06.788 INFO:tasks.rados.rados.0.smithi181.stdout:47: finishing write tid 2 to smithi18128752-47 2024-04-26T01:24:06.825 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:06.827+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:06.831 INFO:tasks.rados.rados.0.smithi181.stdout:52: finishing write tid 1 to smithi18128752-3 2024-04-26T01:24:06.856 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:06.856 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:06.856 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:06.856 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:06.859 INFO:tasks.rados.rados.0.smithi181.stdout:51: finishing copy_from to smithi18128752-2 2024-04-26T01:24:06.859 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-26T01:24:06.859 INFO:tasks.rados.rados.0.smithi181.stdout:25: finishing write tid 3 to smithi18128752-25 2024-04-26T01:24:06.859 INFO:tasks.rados.rados.0.smithi181.stdout:25: oid 25 updating version 0 to 3 2024-04-26T01:24:06.860 INFO:tasks.rados.rados.0.smithi181.stdout:25: oid 25 updating version 3 to 4 2024-04-26T01:24:06.860 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-04-26T01:24:06.860 INFO:tasks.rados.rados.0.smithi181.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-04-26T01:24:06.871 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:24:06.889 INFO:tasks.rados.rados.0.smithi181.stdout:41: finishing write tid 3 to smithi18128752-41 2024-04-26T01:24:06.889 INFO:tasks.rados.rados.0.smithi181.stdout:41: oid 41 updating version 0 to 6 2024-04-26T01:24:06.889 INFO:tasks.rados.rados.0.smithi181.stdout:41: oid 41 version 6 is already newer than 5 2024-04-26T01:24:06.890 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-04-26T01:24:06.890 INFO:tasks.rados.rados.0.smithi181.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-04-26T01:24:06.897 INFO:tasks.rados.rados.0.smithi181.stdout:29: finishing write tid 2 to smithi18128752-29 2024-04-26T01:24:06.926 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:06.928+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:06.932 INFO:tasks.rados.rados.0.smithi181.stdout:52: finishing write tid 2 to smithi18128752-3 2024-04-26T01:24:06.985 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 1 to smithi18128752-48 2024-04-26T01:24:06.986 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 2 to smithi18128752-48 2024-04-26T01:24:06.986 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 3 to smithi18128752-48 2024-04-26T01:24:06.999 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 1 to smithi18128752-49 2024-04-26T01:24:07.026 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:24:07.026 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:07.027+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:07.056 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:07.056 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:07.056 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:07.056 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:07.056 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:07.070 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:24:07.090 INFO:tasks.rados.rados.0.smithi181.stdout:47: finishing write tid 3 to smithi18128752-47 2024-04-26T01:24:07.091 INFO:tasks.rados.rados.0.smithi181.stdout:47: oid 47 updating version 0 to 7 2024-04-26T01:24:07.091 INFO:tasks.rados.rados.0.smithi181.stdout:47: oid 47 updating version 7 to 8 2024-04-26T01:24:07.091 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 47 v 8 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-04-26T01:24:07.091 INFO:tasks.rados.rados.0.smithi181.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-04-26T01:24:07.101 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 2 to smithi18128752-49 2024-04-26T01:24:07.129 INFO:tasks.rados.rados.0.smithi181.stdout:52: finishing write tid 3 to smithi18128752-3 2024-04-26T01:24:07.159 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:07.163+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:07.162 INFO:tasks.rados.rados.0.smithi181.stdout:26: finishing write tid 3 to smithi18128752-26 2024-04-26T01:24:07.162 INFO:tasks.rados.rados.0.smithi181.stdout:26: oid 26 updating version 0 to 5 2024-04-26T01:24:07.162 INFO:tasks.rados.rados.0.smithi181.stdout:26: oid 26 updating version 5 to 6 2024-04-26T01:24:07.162 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-26T01:24:07.162 INFO:tasks.rados.rados.0.smithi181.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-04-26T01:24:07.190 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 4 to smithi18128752-48 2024-04-26T01:24:07.190 INFO:tasks.rados.rados.0.smithi181.stdout:48: oid 48 updating version 0 to 10 2024-04-26T01:24:07.191 INFO:tasks.rados.rados.0.smithi181.stdout:48: oid 48 updating version 10 to 11 2024-04-26T01:24:07.191 INFO:tasks.rados.rados.0.smithi181.stdout:48: oid 48 version 11 is already newer than 9 2024-04-26T01:24:07.191 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 11 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-04-26T01:24:07.191 INFO:tasks.rados.rados.0.smithi181.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-04-26T01:24:07.228 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:07.232+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.6s4 35/34 ECSubRead(tid=8, to_read={3:60f0095a:::smithi18128752-3:head=0,262144,0}, subchunks={3:60f0095a:::smithi18128752-3:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.82596229099999996, 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100734536, 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.241 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420052+0000", 2024-04-26T01:24:07.242 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.846e-06 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420069+0000", 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7493e-05 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420098+0000", 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9079000000000001e-05 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420118+0000", 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9468999999999999e-05 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520369+0000", 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100251189 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520785+0000", 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00041545999999999998 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:07.244 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:07.245 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:07.245 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:07.255 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:25: done (15 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:26: done (14 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:40: done (13 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:41: done (12 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:42: done (11 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:45: done (10 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:47: done (9 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:48: done (8 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:51: done (7 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:54: done (6 left) 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:55: read oid 48 snap -1 2024-04-26T01:24:07.292 INFO:tasks.rados.rados.0.smithi181.stdout:55: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:56: rollback oid 45 current snap is 0 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:56: done (7 left) 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:57: read oid 26 snap -1 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:57: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:58: copy_from oid 44 from oid 18 current snap is 0 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:59: rollback oid 2 current snap is 0 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:59: done (9 left) 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:60: copy_from oid 12 from oid 4 current snap is 0 2024-04-26T01:24:07.293 INFO:tasks.rados.rados.0.smithi181.stdout:61: delete oid 39 current snap is 0 2024-04-26T01:24:07.302 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 3 to smithi18128752-49 2024-04-26T01:24:07.329 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:07.328+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:07.331 INFO:tasks.rados.rados.0.smithi181.stdout:52: finishing write tid 4 to smithi18128752-3 2024-04-26T01:24:07.331 INFO:tasks.rados.rados.0.smithi181.stdout:52: oid 3 updating version 0 to 6 2024-04-26T01:24:07.331 INFO:tasks.rados.rados.0.smithi181.stdout:52: oid 3 version 6 is already newer than 5 2024-04-26T01:24:07.331 INFO:tasks.rados.rados.0.smithi181.stdout:52: oid 3 updating version 6 to 7 2024-04-26T01:24:07.331 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 7 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-26T01:24:07.331 INFO:tasks.rados.rados.0.smithi181.stdout:52: left oid 3 (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s3 36/30 ECSubRead(tid=24, to_read={3:1fe6984d:::smithi18128752-48:head=0,200704,0}, subchunks={3:1fe6984d:::smithi18128752-48:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.299374+0000", 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.098202128999999999, 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098227228999999999, 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.392 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.299374+0000", 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.299374+0000", 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.299375+0000", 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.299387+0000", 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2011000000000001e-05 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.299411+0000", 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3448e-05 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.9s1 36/30 ECSubRead(tid=41, to_read={3:9782f919:::smithi18128752-18:head=0,73728,0}, subchunks={3:9782f919:::smithi18128752-18:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.301793+0000", 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.095783931000000003, 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095841366999999997, 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.301793+0000", 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.301793+0000", 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.301793+0000", 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2399999999999999e-07 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.301804+0000", 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1054e-05 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.301837+0000", 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2872e-05 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.301849+0000", 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1827e-05 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 2 2024-04-26T01:24:07.397 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:07.402 INFO:tasks.rados.rados.0.smithi181.stdout:58: finishing copy_from racing read to smithi18128752-44 2024-04-26T01:24:07.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:24:07.430 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:07.432+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:07.530 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:07.530+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:07.569 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 1 to smithi18128752-50 2024-04-26T01:24:07.569 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:07.569 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:07.569 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:07.569 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:07.569 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:07.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:24:07.606 INFO:tasks.rados.rados.0.smithi181.stdout:60: finishing copy_from racing read to smithi18128752-12 2024-04-26T01:24:07.606 INFO:tasks.rados.rados.0.smithi181.stdout:22: finishing write tid 3 to smithi18128752-22 2024-04-26T01:24:07.606 INFO:tasks.rados.rados.0.smithi181.stdout:22: oid 22 updating version 0 to 7 2024-04-26T01:24:07.606 INFO:tasks.rados.rados.0.smithi181.stdout:22: oid 22 updating version 7 to 8 2024-04-26T01:24:07.606 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-04-26T01:24:07.607 INFO:tasks.rados.rados.0.smithi181.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-04-26T01:24:07.631 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:07.635+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:07.931 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:07.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=9, reqid=client.4589.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.430093+0000", 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.2847276700000001, 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10209124, 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430093+0000", 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430093+0000", 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430095+0000", 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3929999999999998e-06 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430162+0000", 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7224999999999994e-05 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430209+0000", 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6752000000000002e-05 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430236+0000", 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.709e-05 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430384+0000", 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014767599999999999 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.430410+0000", 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5681999999999999e-05 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.532150+0000", 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101740073 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.532184+0000", 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4348999999999999e-05 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.936 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.4s3 35/30 ECSubWrite(tid=10, reqid=client.4589.0:98, at_version=0'0, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.458308+0000", 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.2565124160000001, 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101413427, 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.458308+0000", 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.458308+0000", 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.458309+0000", 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3400000000000001e-06 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.458328+0000", 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8277000000000001e-05 2024-04-26T01:24:07.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.458347+0000", 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8966000000000001e-05 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.458360+0000", 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3193000000000001e-05 2024-04-26T01:24:07.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.558601+0000", 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100241209 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.558623+0000", 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1886999999999999e-05 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.559626+0000", 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001002778 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.559721+0000", 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5777000000000001e-05 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=11, reqid=client.4589.0:127, at_version=35'7, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.136963583, 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202990695, 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577859+0000", 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.013e-06 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578257+0000", 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039834700000000001 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578289+0000", 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2016e-05 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578306+0000", 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6460000000000002e-05 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678641+0000", 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100335199 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678656+0000", 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5233e-05 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780779+0000", 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102122955 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780847+0000", 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8472000000000004e-05 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=12, reqid=client.4589.0:128, at_version=35'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:07.947 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.581325+0000", 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.1334951759999998, 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.199590883, 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581325+0000", 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581325+0000", 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581327+0000", 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9470000000000002e-06 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581393+0000", 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6187999999999997e-05 2024-04-26T01:24:07.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581434+0000", 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1041999999999998e-05 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581493+0000", 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9039000000000002e-05 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779641+0000", 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19814733200000001 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779656+0000", 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.571e-05 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780882+0000", 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012260389999999999 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780916+0000", 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3586e-05 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 35/30 ECSubWrite(tid=167, reqid=client.4589.0:131, at_version=35'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.595107+0000", 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.1197133689999998, 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101581202, 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.952 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.595107+0000", 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.595107+0000", 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.595109+0000", 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.547e-06 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.595141+0000", 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2172000000000003e-05 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.595171+0000", 2024-04-26T01:24:07.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0165999999999999e-05 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.595187+0000", 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6116e-05 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.695440+0000", 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100252909 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.695455+0000", 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5556000000000001e-05 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.696607+0000", 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001151652 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.696688+0000", 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1083999999999998e-05 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.956 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=29, reqid=client.4589.0:118, at_version=35'11, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.631448+0000", 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.08337204, 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15008649800000001, 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631448+0000", 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631448+0000", 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631451+0000", 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.683e-06 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631749+0000", 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000298246 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631785+0000", 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5709e-05 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631812+0000", 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6602999999999999e-05 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779726+0000", 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.147914885 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779743+0000", 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6056999999999999e-05 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.781482+0000", 2024-04-26T01:24:07.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001739167 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.781535+0000", 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3148e-05 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=30, reqid=client.4589.0:119, at_version=35'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.631852+0000", 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.0829685439999999, 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15076103499999999, 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631852+0000", 2024-04-26T01:24:07.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631852+0000", 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631853+0000", 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1999999999999999e-06 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631890+0000", 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6813000000000001e-05 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631911+0000", 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1492999999999999e-05 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631930+0000", 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9162000000000001e-05 2024-04-26T01:24:07.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780836+0000", 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.148905065 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780849+0000", 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3856000000000001e-05 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782457+0000", 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016076300000000001 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782613+0000", 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000155816 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=31, reqid=client.4589.0:124, at_version=35'13, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.86344233, 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20482444299999999, 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851380+0000", 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.973e-06 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852052+0000", 2024-04-26T01:24:07.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067198400000000004 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852084+0000", 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2298999999999997e-05 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952200+0000", 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100116018 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952352+0000", 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015207299999999999 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952377+0000", 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4363e-05 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056116+0000", 2024-04-26T01:24:07.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103738862 2024-04-26T01:24:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056202+0000", 2024-04-26T01:24:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6871000000000002e-05 2024-04-26T01:24:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.973 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=32, reqid=client.4589.0:125, at_version=35'14, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.952255+0000", 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.7625653130000001, 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10401946099999999, 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952255+0000", 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952255+0000", 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952257+0000", 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.516e-06 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952301+0000", 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4019e-05 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952328+0000", 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7236000000000001e-05 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952347+0000", 2024-04-26T01:24:07.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9443e-05 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.953728+0000", 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013808620000000001 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.953755+0000", 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6889e-05 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.977 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056213+0000", 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102457964 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056275+0000", 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1532000000000007e-05 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.978 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=33, reqid=client.4589.0:133, at_version=35'15, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.952388+0000", 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.7624318839999999, 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104377676, 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952388+0000", 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952388+0000", 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952390+0000", 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.11e-06 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952742+0000", 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000352472 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952777+0000", 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4909000000000001e-05 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952802+0000", 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5418999999999998e-05 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.054338+0000", 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101535413 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.054358+0000", 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0533000000000001e-05 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056688+0000", 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023295960000000002 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056766+0000", 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8224000000000002e-05 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=34, reqid=client.4589.0:134, at_version=35'16, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:07.983 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.952844+0000", 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.7619767980000001, 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104441224, 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952844+0000", 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952844+0000", 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952845+0000", 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1030000000000001e-06 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952880+0000", 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5769000000000003e-05 2024-04-26T01:24:07.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952902+0000", 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1849999999999999e-05 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952921+0000", 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8715999999999998e-05 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.055623+0000", 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102701901 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.055650+0000", 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7016e-05 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.057201+0000", 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001551118 2024-04-26T01:24:07.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.057285+0000", 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3751000000000005e-05 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 35/30 ECSubWrite(tid=170, reqid=client.4589.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.745505842, 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20178349000000001, 2024-04-26T01:24:07.988 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.989 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969317+0000", 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0600000000000002e-06 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969365+0000", 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8296e-05 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969439+0000", 2024-04-26T01:24:07.990 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3907999999999998e-05 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969466+0000", 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7223e-05 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169873+0000", 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20040670999999999 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.991 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169887+0000", 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4727e-05 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171018+0000", 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001130968 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171098+0000", 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9598000000000006e-05 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.992 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=13, reqid=client.4589.0:128, at_version=0'0, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.978673+0000", 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.7361476410000001, 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101323183, 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.993 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.978673+0000", 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.978673+0000", 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.978675+0000", 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0660000000000002e-06 2024-04-26T01:24:07.994 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.978744+0000", 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8916000000000002e-05 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.978781+0000", 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7129000000000003e-05 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:07.995 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.078920+0000", 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100139011 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.079060+0000", 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014065099999999999 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.079076+0000", 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5858999999999998e-05 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.996 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.079906+0000", 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00082917600000000004 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.079996+0000", 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0375000000000001e-05 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.997 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 35/34 ECSubWrite(tid=12, reqid=client.4589.0:67, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.175915+0000", 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.5389054049999999, 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105858775, 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.175915+0000", 2024-04-26T01:24:07.998 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.175915+0000", 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.175917+0000", 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.827e-06 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:07.999 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.176193+0000", 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000276586 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.176223+0000", 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9949000000000001e-05 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.276320+0000", 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10009665600000001 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.000 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.276488+0000", 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000168462 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.276516+0000", 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7628999999999999e-05 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.281723+0000", 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0052066880000000001 2024-04-26T01:24:08.001 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.281774+0000", 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0977999999999999e-05 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:08.002 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.437735056, 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20597036399999999, 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:08.003 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277086+0000", 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277250+0000", 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016446899999999999 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.004 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277265+0000", 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.418e-05 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277275+0000", 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0456e-05 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481015+0000", 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203739686 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481042+0000", 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6962999999999999e-05 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482955+0000", 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001913816 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483056+0000", 2024-04-26T01:24:08.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100204 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=15, reqid=client.4589.0:142, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.277533+0000", 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.437287051, 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.181090423, 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277533+0000", 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277533+0000", 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277534+0000", 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8000000000000004e-07 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277731+0000", 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019659199999999999 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277754+0000", 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3612999999999999e-05 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277766+0000", 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1839e-05 2024-04-26T01:24:08.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.378073+0000", 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100307204 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.378088+0000", 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4673e-05 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.458572+0000", 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080483671000000007 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.458624+0000", 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1950999999999998e-05 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.011 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=16, reqid=client.4589.0:143, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.285657+0000", 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.4291635089999999, 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.173905958, 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285657+0000", 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285657+0000", 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285659+0000", 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0159999999999998e-06 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285899+0000", 2024-04-26T01:24:08.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024038099999999999 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285943+0000", 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3849e-05 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.386077+0000", 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100133608 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.457796+0000", 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.071719632000000005 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.457812+0000", 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5982999999999999e-05 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.459449+0000", 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016369519999999999 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.459563+0000", 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011353699999999999 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=37, reqid=client.4589.0:134, at_version=0'0, trim_to=0'0, roll_forward_to=35'16))", 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.380246+0000", 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.3345742170000001, 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.178392667, 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.016 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380246+0000", 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380246+0000", 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380248+0000", 2024-04-26T01:24:08.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.72e-06 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380271+0000", 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3149000000000001e-05 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380301+0000", 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9952999999999999e-05 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380319+0000", 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8312999999999999e-05 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.557548+0000", 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17722879 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.557566+0000", 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8016000000000002e-05 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.558575+0000", 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001008421 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.558639+0000", 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4305000000000002e-05 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.020 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=18, reqid=client.4589.0:144, at_version=0'0, trim_to=0'0, roll_forward_to=35'11))", 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.992135+0000", 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.72268557200000005, 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101421646, 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992135+0000", 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992135+0000", 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992137+0000", 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9599999999999999e-06 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992158+0000", 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1373e-05 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992189+0000", 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1161999999999997e-05 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992206+0000", 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6583e-05 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.092482+0000", 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100276465 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.092499+0000", 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6463000000000001e-05 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093486+0000", 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00098686200000000003 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093556+0000", 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0778000000000006e-05 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.024 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/34 ECSubWrite(tid=12, reqid=client.4589.0:158, at_version=0'0, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.135187+0000", 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.57963341199999996, 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101580092, 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.135187+0000", 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.135187+0000", 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.135189+0000", 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.097e-06 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.135218+0000", 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8881999999999999e-05 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.135262+0000", 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4079000000000003e-05 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.135286+0000", 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3886000000000001e-05 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.235620+0000", 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100333873 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.235639+0000", 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8873e-05 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.236705+0000", 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001065943 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.236767+0000", 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2459000000000006e-05 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.029 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:08.030 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:24:08.030 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 2 to smithi18128752-50 2024-04-26T01:24:08.030 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 3 to smithi18128752-50 2024-04-26T01:24:08.030 INFO:tasks.rados.rados.0.smithi181.stdout:58: finishing copy_from to smithi18128752-44 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:22: done (10 left) 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:52: done (9 left) 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:58: done (8 left) 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:61: done (7 left) 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:62: read oid 21 snap -1 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:62: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:63: snap_create 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:55: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:60: finishing copy_from to smithi18128752-12 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:23: finishing write tid 3 to smithi18128752-23 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:23: oid 23 updating version 0 to 9 2024-04-26T01:24:08.031 INFO:tasks.rados.rados.0.smithi181.stdout:23: oid 23 updating version 9 to 10 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 23 v 10 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:29: finishing write tid 3 to smithi18128752-29 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:29: oid 29 updating version 0 to 11 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:29: oid 29 updating version 11 to 12 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 4 to smithi18128752-49 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:49: oid 49 updating version 0 to 13 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:49: oid 49 updating version 13 to 15 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:49: oid 49 version 15 is already newer than 14 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-04-26T01:24:08.032 INFO:tasks.rados.rados.0.smithi181.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-04-26T01:24:08.033 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:07.930+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:08.033 INFO:tasks.rados.rados.0.smithi181.stdout:62: expect (ObjNum 20 snap 0 seq_num 20) 2024-04-26T01:24:08.033 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:08.035 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:08.036+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:23: done (8 left) 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:29: done (7 left) 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:49: done (6 left) 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:55: done (5 left) 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:60: done (4 left) 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:62: done (3 left) 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:63: done (2 left) 2024-04-26T01:24:08.052 INFO:tasks.rados.rados.0.smithi181.stdout:64: copy_from oid 47 from oid 28 current snap is 1 2024-04-26T01:24:08.053 INFO:tasks.rados.rados.0.smithi181.stdout:65: read oid 36 snap -1 2024-04-26T01:24:08.053 INFO:tasks.rados.rados.0.smithi181.stdout:65: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-26T01:24:08.053 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 39 current snap is 1 2024-04-26T01:24:08.053 INFO:tasks.rados.rados.0.smithi181.stdout:66: seq_num 53 ranges {0=524288,524288=376832} 2024-04-26T01:24:08.060 INFO:tasks.rados.rados.0.smithi181.stdout:66: writing smithi18128752-39 from 0 to 524288 tid 1 2024-04-26T01:24:08.067 INFO:tasks.rados.rados.0.smithi181.stdout:66: writing smithi18128752-39 from 524288 to 901120 tid 2 2024-04-26T01:24:08.067 INFO:tasks.rados.rados.0.smithi181.stdout:67: rollback oid 30 current snap is 1 2024-04-26T01:24:08.067 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 30 to 0 2024-04-26T01:24:08.067 INFO:tasks.rados.rados.0.smithi181.stdout:68: setattr oid 43 current snap is 1 2024-04-26T01:24:08.071 INFO:tasks.rados.rados.0.smithi181.stdout:69: copy_from oid 25 from oid 8 current snap is 1 2024-04-26T01:24:08.071 INFO:tasks.rados.rados.0.smithi181.stdout:70: read oid 31 snap 0 2024-04-26T01:24:08.071 INFO:tasks.rados.rados.0.smithi181.stdout:70: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-26T01:24:08.071 INFO:tasks.rados.rados.0.smithi181.stdout:71: copy_from oid 46 from oid 3 current snap is 1 2024-04-26T01:24:08.071 INFO:tasks.rados.rados.0.smithi181.stdout:72: rollback oid 10 current snap is 1 2024-04-26T01:24:08.071 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 10 to 0 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:73: read oid 24 snap 0 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:73: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:74: rmattr oid 14 current snap is 1 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:74: done (12 left) 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:75: copy_from oid 34 from oid 2 current snap is 1 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:76: read oid 29 snap -1 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:76: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:77: copy_from oid 20 from oid 5 current snap is 1 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:78: read oid 40 snap -1 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout:78: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:24:08.072 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-04-26T01:24:08.080 INFO:tasks.rados.rados.0.smithi181.stdout:57: expect (ObjNum 25 snap 0 seq_num 25) 2024-04-26T01:24:08.112 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 4 to smithi18128752-50 2024-04-26T01:24:08.112 INFO:tasks.rados.rados.0.smithi181.stdout:50: oid 50 updating version 0 to 9 2024-04-26T01:24:08.112 INFO:tasks.rados.rados.0.smithi181.stdout:50: oid 50 version 9 is already newer than 8 2024-04-26T01:24:08.112 INFO:tasks.rados.rados.0.smithi181.stdout:50: oid 50 version 9 is already newer than 7 2024-04-26T01:24:08.112 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-26T01:24:08.112 INFO:tasks.rados.rados.0.smithi181.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-04-26T01:24:08.113 INFO:tasks.rados.rados.0.smithi181.stdout:50: done (15 left) 2024-04-26T01:24:08.113 INFO:tasks.rados.rados.0.smithi181.stdout:57: done (14 left) 2024-04-26T01:24:08.113 INFO:tasks.rados.rados.0.smithi181.stdout:79: setattr oid 12 current snap is 1 2024-04-26T01:24:08.114 INFO:tasks.rados.rados.0.smithi181.stdout:80: snap_create 2024-04-26T01:24:08.135 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:08.134+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:08.168 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:08.168 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:188 3.8s0 3.54728918 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.076662+0000", 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.096661735999999998, 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096729511000000004, 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "op must wait for map", 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 188 2024-04-26T01:24:08.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076662+0000", 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076662+0000", 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.170 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076664+0000", 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4210000000000002e-06 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076807+0000", 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142915 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076828+0000", 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0661000000000001e-05 2024-04-26T01:24:08.171 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076839+0000", 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0806e-05 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076924+0000", 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5341999999999997e-05 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op must wait for map", 2024-04-26T01:24:08.172 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076932+0000", 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0919999999999998e-06 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:191 3.8s0 3.84230328 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.076864+0000", 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.096459330999999995, 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096728804000000002, 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting_for_map not empty", 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 191 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076864+0000", 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076864+0000", 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076865+0000", 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9500000000000002e-07 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076870+0000", 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.181e-06 2024-04-26T01:24:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076880+0000", 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0139999999999999e-05 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076887+0000", 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7889999999999997e-06 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077005+0000", 2024-04-26T01:24:08.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011813 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077011+0000", 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0440000000000001e-06 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:192 3.8s0 3.84230328 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.076900+0000", 2024-04-26T01:24:08.177 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.096423403000000005, 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096803056999999998, 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting_for_map not empty", 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 192 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076900+0000", 2024-04-26T01:24:08.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076900+0000", 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076901+0000", 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2099999999999996e-07 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076905+0000", 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5310000000000003e-06 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076922+0000", 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6993000000000001e-05 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076930+0000", 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.306e-06 2024-04-26T01:24:08.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077045+0000", 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000115277 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077048+0000", 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2890000000000002e-06 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:193 3.8s0 3.84230328 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.076940+0000", 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.096383109999999994, 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096865692000000003, 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting_for_map not empty", 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 193 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.182 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076940+0000", 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076940+0000", 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076941+0000", 2024-04-26T01:24:08.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.1399999999999997e-07 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076954+0000", 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2713e-05 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076962+0000", 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7079999999999997e-06 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.076968+0000", 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0059999999999998e-06 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077071+0000", 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103478 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077076+0000", 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6979999999999997e-06 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.185 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:202 3.8s0 3.b891a298 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.077128+0000", 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.096195567999999995, 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096781403000000002, 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting_for_map not empty", 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.186 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 202 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077128+0000", 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077128+0000", 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077129+0000", 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3790000000000001e-06 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077144+0000", 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4484999999999999e-05 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077166+0000", 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2303e-05 2024-04-26T01:24:08.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077178+0000", 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1943999999999999e-05 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077320+0000", 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014197100000000001 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077325+0000", 2024-04-26T01:24:08.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2750000000000003e-06 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:203 3.8s0 3.b891a298 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.077206+0000", 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.096117641000000004, 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.096806264000000003, 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:08.190 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 203 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077206+0000", 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077206+0000", 2024-04-26T01:24:08.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077207+0000", 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9999999999999995e-07 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077211+0000", 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4469999999999999e-06 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077236+0000", 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4878000000000001e-05 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.077247+0000", 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0883999999999999e-05 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 6 2024-04-26T01:24:08.193 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:08.194 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:24:08.195 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:08.195 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:08.213 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:08.213 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:08.234 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:08.234 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:08.235 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:08.238+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:08.250 INFO:tasks.rados.rados.0.smithi181.stdout:77: finishing copy_from racing read to smithi18128752-20 2024-04-26T01:24:08.274 INFO:tasks.rados.rados.0.smithi181.stdout:66: finishing write tid 1 to smithi18128752-39 2024-04-26T01:24:08.274 INFO:tasks.rados.rados.0.smithi181.stdout:66: finishing write tid 2 to smithi18128752-39 2024-04-26T01:24:08.274 INFO:tasks.rados.rados.0.smithi181.stdout:66: finishing write tid 3 to smithi18128752-39 2024-04-26T01:24:08.318 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:08.318 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:08.318 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:08.318 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:08.318 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:08.332 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:24:08.355 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:08.358+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:08.384 INFO:tasks.rados.rados.0.smithi181.stdout:70: expect (ObjNum 30 snap 0 seq_num 30) 2024-04-26T01:24:08.421 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 5 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T01:24:08.437 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:08.436+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:08.445 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:08.445 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:08.465 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:08.465 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:128 3.8s0 3:18914e2a:::smithi18128752-43:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.575001+0000", 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.8928446340000002, 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.403359048, 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.466 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 128 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575001+0000", 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575001+0000", 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575003+0000", 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.556e-06 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575008+0000", 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6470000000000003e-06 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575023+0000", 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5209e-05 2024-04-26T01:24:08.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575037+0000", 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3329999999999999e-05 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.579797+0000", 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0047605750000000004 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.580048+0000", 2024-04-26T01:24:08.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025018300000000001 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581341+0000", 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012939850000000001 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.775888+0000", 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19454626899999999 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.978121+0000", 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202233204 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.978360+0000", 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023908999999999999 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:129 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:08.471 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.8927402870000001, 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.91360868200000001, 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 129 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575106+0000", 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.043e-06 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575111+0000", 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3370000000000001e-06 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575128+0000", 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7600000000000001e-05 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575138+0000", 2024-04-26T01:24:08.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.877914+0000", 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30277600900000001 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.878073+0000", 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015861299999999999 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979098+0000", 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101025446 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979375+0000", 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000277309 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488714+0000", 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50933870599999997 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:142 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.358204282, 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.88158457700000004, 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 142 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.477 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109644+0000", 2024-04-26T01:24:08.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0970000000000002e-06 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110339+0000", 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00069456900000000002 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110373+0000", 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.358e-05 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110388+0000", 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5726000000000002e-05 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181183+0000", 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.070795134999999995 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181419+0000", 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023538399999999999 2024-04-26T01:24:08.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.182735+0000", 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013160540000000001 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589098+0000", 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40636337 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991098+0000", 2024-04-26T01:24:08.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401999885 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991226+0000", 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012777700000000001 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:143 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:08.482 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.3520366190000002, 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87574223200000001, 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 143 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:08.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115811+0000", 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.317e-06 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116338+0000", 2024-04-26T01:24:08.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052673300000000004 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116375+0000", 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6922999999999999e-05 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116392+0000", 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6629999999999998e-05 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284095+0000", 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16770365000000001 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284265+0000", 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016969199999999999 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285568+0000", 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001302554 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589101+0000", 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30353306000000002 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991441+0000", 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40234051300000001 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991551+0000", 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011016 2024-04-26T01:24:08.487 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:144 3.8s0 3:1fe6984d:::smithi18128752-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.3514262239999999, 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87544766600000001, 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:08.488 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 144 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:08.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116421+0000", 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.15e-06 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116427+0000", 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9070000000000001e-06 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116443+0000", 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-05 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116452+0000", 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0890000000000006e-06 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286741+0000", 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170288786 2024-04-26T01:24:08.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286883+0000", 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142683 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.287610+0000", 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000727172 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589207+0000", 2024-04-26T01:24:08.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30159656099999999 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991768+0000", 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40256125399999998 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991867+0000", 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8814000000000006e-05 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:145 3.8s0 3:1fe6984d:::smithi18128752-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.3513801179999998, 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98038499800000001, 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 145 2024-04-26T01:24:08.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5600000000000004e-07 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116470+0000", 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116484+0000", 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3837e-05 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.216615+0000", 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100131365 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992549+0000", 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77593446499999996 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992720+0000", 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017066299999999999 2024-04-26T01:24:08.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096851+0000", 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104130502 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.1794157169999999, 2024-04-26T01:24:08.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60172694400000004, 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288438+0000", 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1620000000000008e-06 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288450+0000", 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1922999999999999e-05 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288486+0000", 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5905999999999998e-05 2024-04-26T01:24:08.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388601+0000", 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10011487600000001 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890125+0000", 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50152462099999995 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890157+0000", 2024-04-26T01:24:08.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1455999999999998e-05 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.388133+0000", 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.079712695, 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40169646199999998, 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.502 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388133+0000", 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388133+0000", 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388134+0000", 2024-04-26T01:24:08.503 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0999999999999998e-07 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388137+0000", 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.489e-06 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388147+0000", 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0397000000000001e-05 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388155+0000", 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5499999999999997e-06 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.789802+0000", 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40164664999999999 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.789829+0000", 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7766000000000001e-05 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.505 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.388172+0000", 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.079673396, 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40176108999999999, 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388172+0000", 2024-04-26T01:24:08.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388172+0000", 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388173+0000", 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388176+0000", 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.01e-06 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388194+0000", 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8606000000000001e-05 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388208+0000", 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3730000000000001e-05 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.789906+0000", 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40169744499999999 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.789933+0000", 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7832999999999999e-05 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:08.509 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.0791978179999999, 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50193089099999999, 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388649+0000", 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0270000000000001e-06 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388657+0000", 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0260000000000007e-06 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388677+0000", 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0069e-05 2024-04-26T01:24:08.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488777+0000", 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100100183 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890549+0000", 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40177214999999999 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890579+0000", 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9436000000000001e-05 2024-04-26T01:24:08.512 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=15, last_complete=35'9, committed=1, applied=1))", 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.418129+0000", 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.0497170690000002, 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47214447599999998, 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418129+0000", 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418129+0000", 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418130+0000", 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.877e-06 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418140+0000", 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6870000000000006e-06 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418171+0000", 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0479000000000001e-05 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418185+0000", 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:24:08.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890245+0000", 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47205976100000002 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890273+0000", 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7966e-05 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.0496327910000001, 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47216240900000001, 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1399999999999997e-07 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418217+0000", 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.676e-06 2024-04-26T01:24:08.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418228+0000", 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.062e-05 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418235+0000", 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7130000000000005e-06 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890351+0000", 2024-04-26T01:24:08.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47211553299999998 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890375+0000", 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4352999999999999e-05 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.418703+0000", 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 2.0491421829999998, 2024-04-26T01:24:08.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47176985500000002, 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418703+0000", 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418703+0000", 2024-04-26T01:24:08.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418705+0000", 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6440000000000001e-06 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418714+0000", 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2820000000000002e-06 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418744+0000", 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9173e-05 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418758+0000", 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4593000000000001e-05 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890448+0000", 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47169030000000001 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890473+0000", 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4862999999999999e-05 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.978945253, 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50191135200000003, 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.524 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488902+0000", 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9369999999999998e-06 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488913+0000", 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0789e-05 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488954+0000", 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0416000000000001e-05 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589090+0000", 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100136138 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990779+0000", 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401689352 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990812+0000", 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2719999999999998e-05 2024-04-26T01:24:08.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:160 3.8s0 3:1945891d:::smithi18128752-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.9632301170000002, 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.79222307800000002, 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 160 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504618+0000", 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5600000000000001e-06 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504629+0000", 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0956e-05 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504661+0000", 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2428999999999997e-05 2024-04-26T01:24:08.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504678+0000", 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6422999999999998e-05 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093363+0000", 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58868559200000004 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093514+0000", 2024-04-26T01:24:08.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015052300000000001 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.094369+0000", 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00085476900000000001 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096327+0000", 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001957781 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:08.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296743+0000", 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20041606200000001 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296839+0000", 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5983000000000006e-05 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:08.533 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.589137+0000", 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.8787090440000001, 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40179614299999999, 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589137+0000", 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589137+0000", 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589138+0000", 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.22e-06 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589148+0000", 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0263e-05 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589174+0000", 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5876000000000001e-05 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589193+0000", 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8629000000000001e-05 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990901+0000", 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40170817199999997 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990933+0000", 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1983000000000002e-05 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=15, last_complete=35'9, committed=1, applied=1))", 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.589280+0000", 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.878566011, 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40196714900000002, 2024-04-26T01:24:08.537 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589280+0000", 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589280+0000", 2024-04-26T01:24:08.538 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589281+0000", 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589287+0000", 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6329999999999999e-06 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.539 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589310+0000", 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2616000000000002e-05 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589346+0000", 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6479000000000001e-05 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990971+0000", 2024-04-26T01:24:08.540 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40162508699999999 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991247+0000", 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027536699999999999 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.589375+0000", 2024-04-26T01:24:08.541 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.878470737, 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40219637000000003, 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589375+0000", 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.542 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589375+0000", 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589376+0000", 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3699999999999999e-07 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589382+0000", 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2060000000000004e-06 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.543 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589406+0000", 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3966999999999999e-05 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589420+0000", 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4346e-05 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991330+0000", 2024-04-26T01:24:08.544 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401909183 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991571+0000", 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024173099999999999 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.589443+0000", 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.8784030819999999, 2024-04-26T01:24:08.545 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40296986099999998, 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589443+0000", 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589443+0000", 2024-04-26T01:24:08.546 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589443+0000", 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9999999999999996e-07 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589449+0000", 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8370000000000002e-06 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.547 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589469+0000", 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0086e-05 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589492+0000", 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2226e-05 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991643+0000", 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40215106099999998 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.548 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992412+0000", 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00076975100000000005 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:163 3.8s0 3:1fe6984d:::smithi18128752-48:head [read 0~802816,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.297542+0000", 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.1703040929999999, 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40430959399999999, 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:08.549 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 163 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297542+0000", 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.550 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297542+0000", 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297542+0000", 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9599999999999998e-07 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:08.551 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297548+0000", 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4310000000000002e-06 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297567+0000", 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8771e-05 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297576+0000", 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1430000000000004e-06 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:08.552 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.298867+0000", 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001291115 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.298935+0000", 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8131000000000002e-05 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.701851+0000", 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40291620700000003 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.553 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:08.554 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:08.554 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:08.555 INFO:tasks.rados.rados.0.smithi181.stdout:76: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:24:08.555 INFO:tasks.rados.rados.0.smithi181.stdout:66: finishing write tid 4 to smithi18128752-39 2024-04-26T01:24:08.555 INFO:tasks.rados.rados.0.smithi181.stdout:66: oid 39 updating version 0 to 9 2024-04-26T01:24:08.555 INFO:tasks.rados.rados.0.smithi181.stdout:66: oid 39 updating version 9 to 11 2024-04-26T01:24:08.555 INFO:tasks.rados.rados.0.smithi181.stdout:66: oid 39 version 11 is already newer than 10 2024-04-26T01:24:08.556 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 39 v 11 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-04-26T01:24:08.556 INFO:tasks.rados.rados.0.smithi181.stdout:66: left oid 39 (ObjNum 53 snap 1 seq_num 53) 2024-04-26T01:24:08.556 INFO:tasks.rados.rados.0.smithi181.stdout:78: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:24:08.556 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:24:08.557 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:08.559+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:08.557 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 14 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-04-26T01:24:08.557 INFO:tasks.rados.rados.0.smithi181.stdout:73: expect (ObjNum 23 snap 0 seq_num 23) 2024-04-26T01:24:08.571 INFO:tasks.rados.rados.0.smithi181.stdout:69: finishing copy_from racing read to smithi18128752-25 2024-04-26T01:24:08.571 INFO:tasks.rados.rados.0.smithi181.stdout:67: finishing rollback tid 0 to smithi18128752-30 2024-04-26T01:24:08.571 INFO:tasks.rados.rados.0.smithi181.stdout:67: finishing rollback tid 1 to smithi18128752-30 2024-04-26T01:24:08.571 INFO:tasks.rados.rados.0.smithi181.stdout:71: finishing copy_from racing read to smithi18128752-46 2024-04-26T01:24:08.592 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:08.593 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:08.627 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:08.628 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:08.650 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:08.650 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:08.656 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:08.659+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:08.657 INFO:tasks.rados.rados.0.smithi181.stdout:77: finishing copy_from to smithi18128752-20 2024-04-26T01:24:08.657 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T01:24:08.657 INFO:tasks.rados.rados.0.smithi181.stdout:64: finishing copy_from racing read to smithi18128752-47 2024-04-26T01:24:08.664 INFO:tasks.rados.rados.0.smithi181.stdout:65: expect (ObjNum 35 snap 0 seq_num 35) 2024-04-26T01:24:08.684 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:08.685 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:08.690 INFO:tasks.rados.rados.0.smithi181.stdout:71: finishing copy_from to smithi18128752-46 2024-04-26T01:24:08.690 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 46 v 7 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-26T01:24:08.690 INFO:tasks.rados.rados.0.smithi181.stdout:67: finishing rollback tid 2 to smithi18128752-30 2024-04-26T01:24:08.690 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 10 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:24:08.691 INFO:tasks.rados.rados.0.smithi181.stdout:69: finishing copy_from to smithi18128752-25 2024-04-26T01:24:08.691 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 25 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-04-26T01:24:08.691 INFO:tasks.rados.rados.0.smithi181.stdout:72: finishing rollback tid 0 to smithi18128752-10 2024-04-26T01:24:08.691 INFO:tasks.rados.rados.0.smithi181.stdout:72: finishing rollback tid 1 to smithi18128752-10 2024-04-26T01:24:08.691 INFO:tasks.rados.rados.0.smithi181.stdout:72: finishing rollback tid 2 to smithi18128752-10 2024-04-26T01:24:08.691 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 11 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-04-26T01:24:08.705 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:08.705 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:08.726 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:08.726 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:08.734 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:08.734 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:08.747 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:08.747 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:08.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:08.757 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:08.762+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:08.771 INFO:tasks.rados.rados.0.smithi181.stdout:64: finishing copy_from to smithi18128752-47 2024-04-26T01:24:08.771 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 47 v 2 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-04-26T01:24:08.792 INFO:tasks.rados.rados.0.smithi181.stdout:75: finishing copy_from racing read to smithi18128752-34 2024-04-26T01:24:08.814 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:08.815 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:08.837 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:08.838 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:08.857 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:08.862+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:08.866 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:08.866 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:08.875 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:08.875 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:08.889 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:08.919 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:08.920 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:08.952 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:08.953 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:08.958 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:08.960+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:09.014 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:09.015 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:09.029 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:09.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15] out_osds: [7, 12] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:24:09.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:24:09.039 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-04-26T01:24:09.039 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-26T01:24:09.039 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-26T01:24:09.040 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok config set filestore_inject_stall 3 2024-04-26T01:24:09.052 INFO:tasks.rados.rados.0.smithi181.stdout:64: done (15 left) 2024-04-26T01:24:09.052 INFO:tasks.rados.rados.0.smithi181.stdout:65: done (14 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:66: done (13 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:67: done (12 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:68: done (11 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:69: done (10 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:70: done (9 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:71: done (8 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:72: done (7 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:73: done (6 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:76: done (5 left) 2024-04-26T01:24:09.053 INFO:tasks.rados.rados.0.smithi181.stdout:77: done (4 left) 2024-04-26T01:24:09.054 INFO:tasks.rados.rados.0.smithi181.stdout:78: done (3 left) 2024-04-26T01:24:09.054 INFO:tasks.rados.rados.0.smithi181.stdout:79: done (2 left) 2024-04-26T01:24:09.054 INFO:tasks.rados.rados.0.smithi181.stdout:80: done (1 left) 2024-04-26T01:24:09.054 INFO:tasks.rados.rados.0.smithi181.stdout:81: rollback oid 1 current snap is 2 2024-04-26T01:24:09.054 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 1 to 1 2024-04-26T01:24:09.054 INFO:tasks.rados.rados.0.smithi181.stdout:82: snap_create 2024-04-26T01:24:09.054 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:24:09.070 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:09.072+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:09.094 INFO:tasks.rados.rados.0.smithi181.stdout:75: finishing copy_from to smithi18128752-34 2024-04-26T01:24:09.094 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-26T01:24:09.159 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:09.162+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:09.173 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:09.173 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-26T01:24:09.173 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:09.200 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:09.200 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:09.215 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:09.260 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:09.263+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:09.346 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:09.347 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:09.357 INFO:tasks.rados.rados.0.smithi181.stdout:81: finishing rollback tid 0 to smithi18128752-1 2024-04-26T01:24:09.357 INFO:tasks.rados.rados.0.smithi181.stdout:81: finishing rollback tid 1 to smithi18128752-1 2024-04-26T01:24:09.361 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:09.362 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:09.364+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:09.457 INFO:tasks.rados.rados.0.smithi181.stdout:81: finishing rollback tid 2 to smithi18128752-1 2024-04-26T01:24:09.457 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 14 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-26T01:24:09.462 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:09.464+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:09.490 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:09.490 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:09.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:09.563 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:09.564+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:09.635 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:09.635 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:09.649 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:09.663 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:09.667+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:09.765 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:09.766+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:09.818 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:09.818 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:09.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:09.866 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:09.868+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:09.947 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:09.947 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:09.962 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:09.967 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:09.971+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:10.061 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:75: done (2 left) 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:81: done (1 left) 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:82: done (0 left) 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:83: rollback oid 9 current snap is 3 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 9 to 0 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:84: read oid 20 snap 1 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:84: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:85: read oid 41 snap -1 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-26T01:24:10.068 INFO:tasks.rados.rados.0.smithi181.stdout:86: read oid 5 snap -1 2024-04-26T01:24:10.069 INFO:tasks.rados.rados.0.smithi181.stdout:86: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:10.069 INFO:tasks.rados.rados.0.smithi181.stdout:87: read oid 4 snap -1 2024-04-26T01:24:10.069 INFO:tasks.rados.rados.0.smithi181.stdout:87: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:24:10.069 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 3 current snap is 3 2024-04-26T01:24:10.069 INFO:tasks.rados.rados.0.smithi181.stdout:88: seq_num 54 ranges {901120=540672,1441792=376832} 2024-04-26T01:24:10.069 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:10.068+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:10.085 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:10.085 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:10.091 INFO:tasks.rados.rados.0.smithi181.stdout:88: writing smithi18128752-3 from 901120 to 1441792 tid 1 2024-04-26T01:24:10.097 INFO:tasks.rados.rados.0.smithi181.stdout:88: writing smithi18128752-3 from 1441792 to 1818624 tid 2 2024-04-26T01:24:10.097 INFO:tasks.rados.rados.0.smithi181.stdout:89: read oid 27 snap -1 2024-04-26T01:24:10.097 INFO:tasks.rados.rados.0.smithi181.stdout:89: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-26T01:24:10.097 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 17 current snap is 3 2024-04-26T01:24:10.097 INFO:tasks.rados.rados.0.smithi181.stdout:90: seq_num 55 ranges {278528=557056,835584=376832} 2024-04-26T01:24:10.099 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:10.111 INFO:tasks.rados.rados.0.smithi181.stdout:90: writing smithi18128752-17 from 278528 to 835584 tid 1 2024-04-26T01:24:10.117 INFO:tasks.rados.rados.0.smithi181.stdout:90: writing smithi18128752-17 from 835584 to 1212416 tid 2 2024-04-26T01:24:10.117 INFO:tasks.rados.rados.0.smithi181.stdout:91: rollback oid 36 current snap is 3 2024-04-26T01:24:10.117 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 36 to 2 2024-04-26T01:24:10.117 INFO:tasks.rados.rados.0.smithi181.stdout:92: rollback oid 7 current snap is 3 2024-04-26T01:24:10.117 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 7 to 0 2024-04-26T01:24:10.117 INFO:tasks.rados.rados.0.smithi181.stdout:93: snap_create 2024-04-26T01:24:10.169 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:10.171+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:10.265 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:10.266 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:10.269 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:10.271+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:10.280 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:10.370 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:10.374+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:10.400 INFO:tasks.rados.rados.0.smithi181.stdout:89: expect (ObjNum 26 snap 0 seq_num 26) 2024-04-26T01:24:10.406 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:10.406 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:10.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:10.434 INFO:tasks.rados.rados.0.smithi181.stdout:92: finishing rollback tid 0 to smithi18128752-7 2024-04-26T01:24:10.434 INFO:tasks.rados.rados.0.smithi181.stdout:92: finishing rollback tid 1 to smithi18128752-7 2024-04-26T01:24:10.472 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:10.478+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:10.500 INFO:tasks.rados.rados.0.smithi181.stdout:83: finishing rollback tid 0 to smithi18128752-9 2024-04-26T01:24:10.500 INFO:tasks.rados.rados.0.smithi181.stdout:83: finishing rollback tid 1 to smithi18128752-9 2024-04-26T01:24:10.526 INFO:tasks.rados.rados.0.smithi181.stdout:92: finishing rollback tid 2 to smithi18128752-7 2024-04-26T01:24:10.526 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 7 v 11 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-26T01:24:10.538 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:10.538 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:10.552 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:10.573 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:10.574+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:83: finishing rollback tid 2 to smithi18128752-9 2024-04-26T01:24:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 18 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-04-26T01:24:10.604 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 1 to smithi18128752-3 2024-04-26T01:24:10.604 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 2 to smithi18128752-3 2024-04-26T01:24:10.604 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 3 to smithi18128752-3 2024-04-26T01:24:10.607 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 4 to smithi18128752-3 2024-04-26T01:24:10.607 INFO:tasks.rados.rados.0.smithi181.stdout:88: oid 3 updating version 0 to 13 2024-04-26T01:24:10.607 INFO:tasks.rados.rados.0.smithi181.stdout:88: oid 3 updating version 13 to 14 2024-04-26T01:24:10.607 INFO:tasks.rados.rados.0.smithi181.stdout:88: oid 3 updating version 14 to 15 2024-04-26T01:24:10.607 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 15 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-04-26T01:24:10.607 INFO:tasks.rados.rados.0.smithi181.stdout:88: left oid 3 (ObjNum 54 snap 3 seq_num 54) 2024-04-26T01:24:10.670 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:10.670 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:10.674 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:10.676+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:10.676 INFO:tasks.rados.rados.0.smithi181.stdout:84: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:10.685 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:10.775 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:10.780+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:10.786 INFO:tasks.rados.rados.0.smithi181.stdout:90: finishing write tid 1 to smithi18128752-17 2024-04-26T01:24:10.796 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:10.796 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:10.809 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:10.876 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:10.880+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:10.886 INFO:tasks.rados.rados.0.smithi181.stdout:90: finishing write tid 2 to smithi18128752-17 2024-04-26T01:24:10.917 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:10.917 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:10.930 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:10.977 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:10.979+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:10.980 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:11.074 INFO:tasks.rados.rados.0.smithi181.stdout:83: done (10 left) 2024-04-26T01:24:11.075 INFO:tasks.rados.rados.0.smithi181.stdout:84: done (9 left) 2024-04-26T01:24:11.075 INFO:tasks.rados.rados.0.smithi181.stdout:88: done (8 left) 2024-04-26T01:24:11.075 INFO:tasks.rados.rados.0.smithi181.stdout:89: done (7 left) 2024-04-26T01:24:11.075 INFO:tasks.rados.rados.0.smithi181.stdout:92: done (6 left) 2024-04-26T01:24:11.075 INFO:tasks.rados.rados.0.smithi181.stdout:93: done (5 left) 2024-04-26T01:24:11.075 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 23 current snap is 4 2024-04-26T01:24:11.075 INFO:tasks.rados.rados.0.smithi181.stdout:94: seq_num 56 ranges {376832=573440,950272=376832} 2024-04-26T01:24:11.077 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:11.077+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:11.087 INFO:tasks.rados.rados.0.smithi181.stdout:94: writing smithi18128752-23 from 376832 to 950272 tid 1 2024-04-26T01:24:11.092 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:11.092 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:11.093 INFO:tasks.rados.rados.0.smithi181.stdout:94: writing smithi18128752-23 from 950272 to 1327104 tid 2 2024-04-26T01:24:11.093 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 22 current snap is 4 2024-04-26T01:24:11.093 INFO:tasks.rados.rados.0.smithi181.stdout:95: seq_num 57 ranges {360448=589824,950272=376832} 2024-04-26T01:24:11.106 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:11.107 INFO:tasks.rados.rados.0.smithi181.stdout:95: writing smithi18128752-22 from 360448 to 950272 tid 1 2024-04-26T01:24:11.112 INFO:tasks.rados.rados.0.smithi181.stdout:95: writing smithi18128752-22 from 950272 to 1327104 tid 2 2024-04-26T01:24:11.112 INFO:tasks.rados.rados.0.smithi181.stdout:96: read oid 33 snap -1 2024-04-26T01:24:11.112 INFO:tasks.rados.rados.0.smithi181.stdout:96: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:11.112 INFO:tasks.rados.rados.0.smithi181.stdout:97: rollback oid 12 current snap is 4 2024-04-26T01:24:11.112 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 12 to 3 2024-04-26T01:24:11.113 INFO:tasks.rados.rados.0.smithi181.stdout:91: finishing rollback tid 0 to smithi18128752-36 2024-04-26T01:24:11.113 INFO:tasks.rados.rados.0.smithi181.stdout:91: finishing rollback tid 1 to smithi18128752-36 2024-04-26T01:24:11.113 INFO:tasks.rados.rados.0.smithi181.stdout:90: finishing write tid 3 to smithi18128752-17 2024-04-26T01:24:11.113 INFO:tasks.rados.rados.0.smithi181.stdout:98: snap_remove snap 0 2024-04-26T01:24:11.144 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:11.145 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:11.165 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:11.166 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:11.178 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:11.180+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:11.186 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:11.186 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:11.208 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:11.208 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:11.229 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:11.229 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:11.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:11.284 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:11.288+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:11.312 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:11.312 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:11.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:11.343 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:11.374 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.306993+0000", 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.073280253000000004, 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.073321226000000003, 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.306993+0000", 2024-04-26T01:24:11.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.306993+0000", 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.306994+0000", 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.017e-06 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307000+0000", 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6879999999999996e-06 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307016+0000", 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5716e-05 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307024+0000", 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3799999999999994e-06 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.073217011999999998, 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.073346291999999993, 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307065+0000", 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9039999999999992e-06 2024-04-26T01:24:11.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307072+0000", 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5619999999999998e-06 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307078+0000", 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1830000000000002e-06 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.380 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.054236312000000002, 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.054443393999999999, 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326038+0000", 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0920000000000001e-06 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326046+0000", 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2339999999999997e-06 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326060+0000", 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4187e-05 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326083+0000", 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3640000000000001e-05 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:11.383 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.051288548000000003, 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.051572527999999999, 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:11.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328986+0000", 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0839999999999999e-06 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328992+0000", 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8130000000000001e-06 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.329007+0000", 2024-04-26T01:24:11.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4769e-05 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.329014+0000", 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4240000000000004e-06 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.15451975800000001, 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.15487985600000001, 2024-04-26T01:24:11.386 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225756+0000", 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.756e-06 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225765+0000", 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225784+0000", 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-04-26T01:24:11.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.325876+0000", 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100092051 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 5 2024-04-26T01:24:11.389 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:11.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:11.391 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:11.392+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:11.453 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:11.454 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:11.484 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:11.485 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:11.491 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:11.492+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:11.515 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:11.516 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:11.516 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:11.516 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:11.516 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:11.520 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:11.520 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:11.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:11.555 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:11.555 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:11.584 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:11.585 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:11.592 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:11.596+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:11.612 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:11.613 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:11.640 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:11.640 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:11.654 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:11.654 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:11.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:11.654 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:11.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.655 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s6 35/34 ECSubWrite(tid=165, reqid=client.4589.0:113, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:11.655 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:05.384986+0000", 2024-04-26T01:24:11.655 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.2700504050000001, 2024-04-26T01:24:11.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10190149900000001, 2024-04-26T01:24:11.655 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.655 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.384986+0000", 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.384986+0000", 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.384988+0000", 2024-04-26T01:24:11.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7e-06 2024-04-26T01:24:11.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.385019+0000", 2024-04-26T01:24:11.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1606000000000002e-05 2024-04-26T01:24:11.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.385045+0000", 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6356e-05 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.385068+0000", 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3005000000000001e-05 2024-04-26T01:24:11.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.485605+0000", 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100536415 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:11.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.485621+0000", 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5962999999999999e-05 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.486809+0000", 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011885660000000001 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.486887+0000", 2024-04-26T01:24:11.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7887999999999997e-05 2024-04-26T01:24:11.662 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.662 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.662 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 35/30 ECSubWrite(tid=37, reqid=client.4589.0:121, at_version=35'5, trim_to=0'0, roll_forward_to=34'4))", 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:05.596405+0000", 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0586311390000001, 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102485587, 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596405+0000", 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596405+0000", 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596407+0000", 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.804e-06 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596661+0000", 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025394000000000003 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596684+0000", 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3278999999999999e-05 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.696819+0000", 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100134475 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.696987+0000", 2024-04-26T01:24:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016887299999999999 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.697009+0000", 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1081999999999999e-05 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.698821+0000", 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018124619999999999 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.698891+0000", 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9672000000000006e-05 2024-04-26T01:24:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 35/30 ECSubWrite(tid=42, reqid=client.4589.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.086757+0000", 2024-04-26T01:24:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.56827892, 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101553475, 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086757+0000", 2024-04-26T01:24:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086757+0000", 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086759+0000", 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7129999999999999e-06 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086779+0000", 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0403000000000001e-05 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086809+0000", 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9085999999999999e-05 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186953+0000", 2024-04-26T01:24:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100144019 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.187126+0000", 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000173105 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.187154+0000", 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8218999999999999e-05 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.188230+0000", 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010764279999999999 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.188311+0000", 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0501999999999996e-05 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 35/30 ECSubWrite(tid=47, reqid=client.4589.0:140, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.292484+0000", 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3625524799999997, 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10087917, 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.292484+0000", 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.292484+0000", 2024-04-26T01:24:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.292485+0000", 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7230000000000001e-06 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.292520+0000", 2024-04-26T01:24:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4326000000000003e-05 2024-04-26T01:24:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.292549+0000", 2024-04-26T01:24:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9173e-05 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.292564+0000", 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5478999999999999e-05 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.392207+0000", 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099642306 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.392237+0000", 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0326000000000001e-05 2024-04-26T01:24:11.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:11.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.393305+0000", 2024-04-26T01:24:11.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001068418 2024-04-26T01:24:11.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.393363+0000", 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7419e-05 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:169 3.as0 3:56e8b2fd:::smithi18128752-12:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.3570608499999999, 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.415445014, 2024-04-26T01:24:11.685 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.686 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.686 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:11.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:11.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:11.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:11.686 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 169 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297980+0000", 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3359999999999997e-06 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297998+0000", 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8499999999999999e-05 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298041+0000", 2024-04-26T01:24:11.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3171999999999999e-05 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398202+0000", 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10016098800000001 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398693+0000", 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049036800000000003 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398913+0000", 2024-04-26T01:24:11.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220051 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.610194+0000", 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.211280998 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612042+0000", 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018484860000000001 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:11.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713242+0000", 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10119953 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713420+0000", 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017858500000000001 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:170 3.as0 3:56e8b2fd:::smithi18128752-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:11.691 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.2567960679999999, 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.213653854, 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 170 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398242+0000", 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0530000000000001e-06 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398252+0000", 2024-04-26T01:24:11.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4830000000000005e-06 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398271+0000", 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-05 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398282+0000", 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1476e-05 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404421+0000", 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006138592 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404504+0000", 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2750999999999994e-05 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611404+0000", 2024-04-26T01:24:11.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20690029200000001 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611712+0000", 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030829799999999998 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611894+0000", 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018167899999999999 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.696 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/30 ECSubWriteReply(tid=4, last_complete=36'3, committed=1, applied=1))", 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.612433+0000", 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0426031939999998, 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10031192899999999, 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.697 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612433+0000", 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612433+0000", 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612434+0000", 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7000000000000003e-07 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612440+0000", 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9100000000000002e-06 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612453+0000", 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3339000000000001e-05 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612493+0000", 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9292999999999998e-05 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.712711+0000", 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10021799100000001 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.712745+0000", 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4425999999999999e-05 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.700 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/30 ECSubWriteReply(tid=4, last_complete=36'3, committed=1, applied=1))", 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.612579+0000", 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0424568079999998, 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100275213, 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612579+0000", 2024-04-26T01:24:11.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612579+0000", 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612581+0000", 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4169999999999999e-06 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612588+0000", 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3529999999999997e-06 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612606+0000", 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8309999999999999e-05 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612618+0000", 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1549000000000001e-05 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.712827+0000", 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10020862799999999 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.712855+0000", 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7956000000000001e-05 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/30 ECSubWriteReply(tid=4, last_complete=36'3, committed=1, applied=1))", 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.612653+0000", 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0423830499999998, 2024-04-26T01:24:11.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100341705, 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612653+0000", 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612653+0000", 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612656+0000", 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6699999999999998e-06 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612668+0000", 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1933000000000001e-05 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612701+0000", 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2906e-05 2024-04-26T01:24:11.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612721+0000", 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0109999999999999e-05 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.712954+0000", 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100233513 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.712995+0000", 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0572999999999999e-05 2024-04-26T01:24:11.707 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 36/30 ECSubWriteReply(tid=5, last_complete=36'3, committed=1, applied=1))", 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.715628+0000", 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.9394086619999999, 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099839664999999994, 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.715628+0000", 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.715628+0000", 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.715628+0000", 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2999999999999999e-07 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.715633+0000", 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5600000000000004e-06 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.715645+0000", 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2023e-05 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.715654+0000", 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6260000000000001e-06 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.815431+0000", 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099777481000000001 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.815467+0000", 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6145e-05 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.711 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:212 3.as0 3:56e8b2fd:::smithi18128752-12:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.5340374190000001, 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.306234695, 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:11.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121002+0000", 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.366e-06 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121071+0000", 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9202000000000006e-05 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121106+0000", 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5052999999999998e-05 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121125+0000", 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8219e-05 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221415+0000", 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029012800000001 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221427+0000", 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.187e-05 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222099+0000", 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067277899999999998 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222322+0000", 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000222272 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.224059+0000", 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017374059999999999 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226116+0000", 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020565679999999999 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426988+0000", 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20087223500000001 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.427234+0000", 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024559700000000001 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 37/30 ECSubWriteReply(tid=6, last_complete=37'5, committed=1, applied=1))", 2024-04-26T01:24:11.717 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.226125+0000", 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.4289107469999998, 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.158071511, 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226125+0000", 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226125+0000", 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226128+0000", 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4870000000000001e-06 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226140+0000", 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1566000000000001e-05 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226171+0000", 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1316000000000003e-05 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326267+0000", 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009647000000001 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384168+0000", 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057900419000000002 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384197+0000", 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9252999999999999e-05 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 37/30 ECSubWriteReply(tid=6, last_complete=37'5, committed=1, applied=1))", 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.277623+0000", 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.3774130749999998, 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10031179, 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.277623+0000", 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.277623+0000", 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.277625+0000", 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.906e-06 2024-04-26T01:24:11.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.277635+0000", 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6029999999999994e-06 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.277693+0000", 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7964999999999998e-05 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.277710+0000", 2024-04-26T01:24:11.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7560000000000001e-05 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.377905+0000", 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10019478699999999 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.377935+0000", 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9969000000000001e-05 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 37/30 ECSubWriteReply(tid=6, last_complete=37'5, committed=1, applied=1))", 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.326457+0000", 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.32857878, 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101657281, 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326457+0000", 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326457+0000", 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326459+0000", 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6670000000000001e-06 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326467+0000", 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9160000000000001e-06 2024-04-26T01:24:11.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326485+0000", 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8050000000000002e-05 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426618+0000", 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100133342 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426762+0000", 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014365299999999999 2024-04-26T01:24:11.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.428115+0000", 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013526530000000001 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.42928245599999998, 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28191508700000001, 2024-04-26T01:24:11.728 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225756+0000", 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.756e-06 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225765+0000", 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225784+0000", 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.325876+0000", 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100092051 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507611+0000", 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.181735758 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507669+0000", 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7361999999999997e-05 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.731 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.306993+0000", 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.34804295099999999, 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029036199999999, 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.306993+0000", 2024-04-26T01:24:11.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.306993+0000", 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.306994+0000", 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.017e-06 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307000+0000", 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6879999999999996e-06 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307016+0000", 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5716e-05 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307024+0000", 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3799999999999994e-06 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.407224+0000", 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10020033 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.407284+0000", 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9231000000000003e-05 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.34797971, 2024-04-26T01:24:11.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044763199999999, 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307065+0000", 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9039999999999992e-06 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307072+0000", 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5619999999999998e-06 2024-04-26T01:24:11.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307078+0000", 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1830000000000002e-06 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507467+0000", 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200389598 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507504+0000", 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6879999999999999e-05 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.32899900999999998, 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.182176437, 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326038+0000", 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0920000000000001e-06 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326046+0000", 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2339999999999997e-06 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326060+0000", 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4187e-05 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326083+0000", 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3640000000000001e-05 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507750+0000", 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18166618600000001 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.508214+0000", 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046409799999999997 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:11.742 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.32605124600000002, 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19916183800000001, 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:11.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328986+0000", 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0839999999999999e-06 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328992+0000", 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8130000000000001e-06 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.329007+0000", 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4769e-05 2024-04-26T01:24:11.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.329014+0000", 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4240000000000004e-06 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.528083+0000", 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19906906299999999 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.528147+0000", 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3684999999999998e-05 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.745 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.427744+0000", 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.22729222199999999, 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100521285, 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.427744+0000", 2024-04-26T01:24:11.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.427744+0000", 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.427746+0000", 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0540000000000002e-06 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.427753+0000", 2024-04-26T01:24:11.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5599999999999999e-06 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.427772+0000", 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9803999999999999e-05 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.527918+0000", 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014508499999999 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.528218+0000", 2024-04-26T01:24:11.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030040600000000002 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.528265+0000", 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7376000000000002e-05 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:11.749 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:11.750 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:11.751 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:11.751 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:11.752 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:11.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:11.752 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:11.753 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:11.754 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:11.756+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:11.852 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:11.856+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:11.880 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:11.880 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:11.880 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:11.881 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:11.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:11.953 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:11.956+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:12.017 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:12.017 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:12.017 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:12.017 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:12.017 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:12.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:24:12.054 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:12.056+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:12.088 INFO:tasks.rados.rados.0.smithi181.stdout:98: done (9 left) 2024-04-26T01:24:12.088 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 43 current snap is 4 2024-04-26T01:24:12.088 INFO:tasks.rados.rados.0.smithi181.stdout:85: expect (ObjNum 40 snap 0 seq_num 40) 2024-04-26T01:24:12.088 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:24:12.114 INFO:tasks.rados.rados.0.smithi181.stdout:86: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:12.117 INFO:tasks.rados.rados.0.smithi181.stdout:97: finishing rollback tid 0 to smithi18128752-12 2024-04-26T01:24:12.117 INFO:tasks.rados.rados.0.smithi181.stdout:91: finishing rollback tid 2 to smithi18128752-36 2024-04-26T01:24:12.117 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 14 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-04-26T01:24:12.117 INFO:tasks.rados.rados.0.smithi181.stdout:87: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:24:12.119 INFO:tasks.rados.rados.0.smithi181.stdout:90: finishing write tid 4 to smithi18128752-17 2024-04-26T01:24:12.120 INFO:tasks.rados.rados.0.smithi181.stdout:90: oid 17 updating version 0 to 16 2024-04-26T01:24:12.120 INFO:tasks.rados.rados.0.smithi181.stdout:90: oid 17 updating version 16 to 18 2024-04-26T01:24:12.120 INFO:tasks.rados.rados.0.smithi181.stdout:90: oid 17 version 18 is already newer than 17 2024-04-26T01:24:12.120 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 18 (ObjNum 55 snap 3 seq_num 55) dirty exists 2024-04-26T01:24:12.120 INFO:tasks.rados.rados.0.smithi181.stdout:90: left oid 17 (ObjNum 55 snap 3 seq_num 55) 2024-04-26T01:24:12.120 INFO:tasks.rados.rados.0.smithi181.stdout:96: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:12.139 INFO:tasks.rados.rados.0.smithi181.stdout:97: finishing rollback tid 1 to smithi18128752-12 2024-04-26T01:24:12.139 INFO:tasks.rados.rados.0.smithi181.stdout:94: finishing write tid 1 to smithi18128752-23 2024-04-26T01:24:12.139 INFO:tasks.rados.rados.0.smithi181.stdout:94: finishing write tid 2 to smithi18128752-23 2024-04-26T01:24:12.139 INFO:tasks.rados.rados.0.smithi181.stdout:94: finishing write tid 3 to smithi18128752-23 2024-04-26T01:24:12.139 INFO:tasks.rados.rados.0.smithi181.stdout:97: finishing rollback tid 2 to smithi18128752-12 2024-04-26T01:24:12.139 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 9 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T01:24:12.140 INFO:tasks.rados.rados.0.smithi181.stdout:95: finishing write tid 1 to smithi18128752-22 2024-04-26T01:24:12.140 INFO:tasks.rados.rados.0.smithi181.stdout:95: finishing write tid 2 to smithi18128752-22 2024-04-26T01:24:12.140 INFO:tasks.rados.rados.0.smithi181.stdout:99: seq_num 58 ranges {720896=606208,1327104=376832} 2024-04-26T01:24:12.155 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:12.157+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:12.157 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:12.158 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:12.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:12.158 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2024-04-26T01:24:12.158 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:12.160 INFO:tasks.rados.rados.0.smithi181.stdout:99: writing smithi18128752-43 from 720896 to 1327104 tid 1 2024-04-26T01:24:12.167 INFO:tasks.rados.rados.0.smithi181.stdout:99: writing smithi18128752-43 from 1327104 to 1703936 tid 2 2024-04-26T01:24:12.167 INFO:tasks.rados.rados.0.smithi181.stdout:85: done (9 left) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:86: done (8 left) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:87: done (7 left) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:90: done (6 left) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:91: done (5 left) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:96: done (4 left) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:97: done (3 left) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:100: read oid 28 snap -1 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:100: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:101: snap_create 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:94: finishing write tid 4 to smithi18128752-23 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:94: oid 23 updating version 0 to 21 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:94: oid 23 version 21 is already newer than 19 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:94: oid 23 version 21 is already newer than 20 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 23 v 21 (ObjNum 56 snap 4 seq_num 56) dirty exists 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:94: left oid 23 (ObjNum 56 snap 4 seq_num 56) 2024-04-26T01:24:12.168 INFO:tasks.rados.rados.0.smithi181.stdout:95: finishing write tid 3 to smithi18128752-22 2024-04-26T01:24:12.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:24:12.256 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:12.256+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:12.287 INFO:tasks.rados.rados.0.smithi181.stdout:95: finishing write tid 4 to smithi18128752-22 2024-04-26T01:24:12.287 INFO:tasks.rados.rados.0.smithi181.stdout:95: oid 22 updating version 0 to 24 2024-04-26T01:24:12.288 INFO:tasks.rados.rados.0.smithi181.stdout:95: oid 22 version 24 is already newer than 23 2024-04-26T01:24:12.288 INFO:tasks.rados.rados.0.smithi181.stdout:95: oid 22 version 24 is already newer than 22 2024-04-26T01:24:12.288 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 24 (ObjNum 57 snap 4 seq_num 57) dirty exists 2024-04-26T01:24:12.288 INFO:tasks.rados.rados.0.smithi181.stdout:95: left oid 22 (ObjNum 57 snap 4 seq_num 57) 2024-04-26T01:24:12.339 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:12.339 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:12.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.339 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.as6 41/30 ECSubWrite(tid=12, reqid=osd.3.0:12, at_version=41'11, trim_to=0'0, roll_forward_to=40'9))", 2024-04-26T01:24:12.339 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.302410+0000", 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.042834103999999998, 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.042873300000000003, 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.302410+0000", 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.302410+0000", 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.302410+0000", 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.36e-07 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.302433+0000", 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2565000000000002e-05 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.302447+0000", 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4469e-05 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2024-04-26T01:24:12.342 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:12.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:24:12.357 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:12.359+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:12.458 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:12.459+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:12.477 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:12.477 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:12.477 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:12.477 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:12.477 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:12.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:24:12.558 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:12.564+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:12.581 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 1 to smithi18128752-43 2024-04-26T01:24:12.582 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 2 to smithi18128752-43 2024-04-26T01:24:12.582 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 3 to smithi18128752-43 2024-04-26T01:24:12.619 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:12.619 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:12.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:12.619 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:12.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.619 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:150 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.4427102859999996, 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3959368480000001, 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 150 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179064+0000", 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7269999999999999e-06 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179608+0000", 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00054408799999999997 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179641+0000", 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2873000000000002e-05 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179658+0000", 2024-04-26T01:24:12.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6863000000000001e-05 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268330+0000", 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0886726819999999 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268562+0000", 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023123100000000001 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.270316+0000", 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017541189999999999 2024-04-26T01:24:12.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474438+0000", 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20412262 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574871+0000", 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100432788 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574998+0000", 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126857 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.624 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:151 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.4363354240000001, 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.494027684, 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 151 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185438+0000", 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2170000000000001e-06 2024-04-26T01:24:12.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185895+0000", 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000456291 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185927+0000", 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1989000000000002e-05 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185942+0000", 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5075999999999999e-05 2024-04-26T01:24:12.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272106+0000", 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0861648319999999 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272243+0000", 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013630399999999999 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.273909+0000", 2024-04-26T01:24:12.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016665930000000001 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474512+0000", 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20060244399999999 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679353+0000", 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20484124000000001 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679464+0000", 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110698 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:152 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.4358035180000002, 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4937803999999999, 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.630 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 152 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185969+0000", 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.167e-06 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185975+0000", 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5029999999999997e-06 2024-04-26T01:24:12.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185986+0000", 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1283e-05 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185994+0000", 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6060000000000003e-06 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372230+0000", 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.186235766 2024-04-26T01:24:12.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372376+0000", 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014574 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.373451+0000", 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010752890000000001 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474514+0000", 2024-04-26T01:24:12.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101062945 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679632+0000", 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20511794799999999 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679748+0000", 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116153 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.635 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:153 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.4357651420000002, 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.900776316, 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 153 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186007+0000", 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6700000000000002e-07 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186011+0000", 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5559999999999999e-06 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186025+0000", 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3873000000000001e-05 2024-04-26T01:24:12.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186035+0000", 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0526000000000001e-05 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475079+0000", 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2890438200000001 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475255+0000", 2024-04-26T01:24:12.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000176279 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680464+0000", 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20520917499999999 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680595+0000", 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013023099999999999 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086783+0000", 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40618798900000003 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 35/34 ECSubWriteReply(tid=16, last_complete=35'6, committed=1, applied=1))", 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.458536+0000", 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1632358580000002, 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30518046599999998, 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.641 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458536+0000", 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458536+0000", 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458536+0000", 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7599999999999998e-07 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458539+0000", 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0199999999999999e-06 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458548+0000", 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.653e-06 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458554+0000", 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5740000000000001e-06 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.763685+0000", 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30513032699999998 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.763716+0000", 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1516000000000001e-05 2024-04-26T01:24:12.644 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1631082780000002, 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.407182076, 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458664+0000", 2024-04-26T01:24:12.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458680+0000", 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5953000000000001e-05 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458693+0000", 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2972999999999999e-05 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458701+0000", 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4829999999999999e-06 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.865517+0000", 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40681550500000002 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.865845+0000", 2024-04-26T01:24:12.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032856200000000002 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 35/34 ECSubWriteReply(tid=16, last_complete=35'6, committed=1, applied=1))", 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.458759+0000", 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1630124669999997, 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.305802193, 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.649 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458759+0000", 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458759+0000", 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458759+0000", 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4000000000000002e-07 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458763+0000", 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3359999999999999e-06 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458772+0000", 2024-04-26T01:24:12.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3330000000000003e-06 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458779+0000", 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1369999999999996e-06 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.763783+0000", 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.305004161 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.764561+0000", 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077778600000000004 2024-04-26T01:24:12.652 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.459055+0000", 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1627163649999996, 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30627369100000001, 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.459055+0000", 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.459055+0000", 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.459056+0000", 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.459069+0000", 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3339000000000001e-05 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.459080+0000", 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0497e-05 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.459088+0000", 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8200000000000003e-06 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.765263+0000", 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30617506 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.765329+0000", 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5384999999999998e-05 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:164 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:12.656 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3238586799999998, 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78562828299999998, 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:12.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297916+0000", 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297932+0000", 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2542e-05 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297993+0000", 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7813e-05 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475329+0000", 2024-04-26T01:24:12.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17733589199999999 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475446+0000", 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117074 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083541+0000", 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60809543499999996 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:165 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3237432660000001, 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78769006399999997, 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298030+0000", 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.59e-06 2024-04-26T01:24:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298039+0000", 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1470000000000007e-06 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298064+0000", 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5542e-05 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298080+0000", 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5279999999999999e-05 2024-04-26T01:24:12.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476021+0000", 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.177941503 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476177+0000", 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155873 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085718+0000", 2024-04-26T01:24:12.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60954112900000001 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:166 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3236672250000003, 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78831037500000001, 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.665 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 166 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:12.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298106+0000", 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-06 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298113+0000", 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7570000000000004e-06 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298128+0000", 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5e-05 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298139+0000", 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0986e-05 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476693+0000", 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17855336099999999 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476835+0000", 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014191599999999999 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086415+0000", 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60957998499999999 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/34 ECSubReadReply(tid=23, attrs_read=0))", 2024-04-26T01:24:12.669 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.0211491989999999, 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.48295446400000003, 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600624+0000", 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6199999999999999e-06 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601039+0000", 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041512500000000002 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601068+0000", 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.906e-05 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701180+0000", 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10011215699999999 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.082862+0000", 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.381681457 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083577+0000", 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071504499999999998 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/34 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.701229+0000", 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9205423650000002, 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.38452293799999998, 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.673 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701229+0000", 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701229+0000", 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701230+0000", 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.277e-06 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701365+0000", 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013504000000000001 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701389+0000", 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3558999999999999e-05 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701400+0000", 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1420000000000001e-05 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085109+0000", 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.383708365 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085752+0000", 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00064327700000000004 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.676 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/34 ECSubReadReply(tid=25, attrs_read=0))", 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.701418+0000", 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9203535629999999, 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.38502355599999999, 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701418+0000", 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701418+0000", 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701419+0000", 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.16e-07 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701661+0000", 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024235399999999999 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701691+0000", 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9796000000000001e-05 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701711+0000", 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0092999999999999e-05 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085878+0000", 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.38416697799999999 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086441+0000", 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056351899999999996 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/34 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.701896+0000", 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.919875609, 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.383174503, 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701896+0000", 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701896+0000", 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701896+0000", 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1699999999999998e-07 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701981+0000", 2024-04-26T01:24:12.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4340999999999998e-05 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701994+0000", 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3169999999999999e-05 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.802138+0000", 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014419099999999 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.084975+0000", 2024-04-26T01:24:12.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28283637 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085070+0000", 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5814000000000001e-05 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:189 3.cs0 3:395c8678:::smithi18128752-25:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.5447316129999997, 2024-04-26T01:24:12.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61527076999999997, 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077044+0000", 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0570000000000003e-06 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077061+0000", 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6616000000000002e-05 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077097+0000", 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6829000000000003e-05 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077120+0000", 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2779999999999999e-05 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086851+0000", 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097310599999999997 2024-04-26T01:24:12.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.087007+0000", 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015570199999999999 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.495435+0000", 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40842789299999999 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497946+0000", 2024-04-26T01:24:12.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025112519999999998 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692090+0000", 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19414421300000001 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692311+0000", 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220368 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:190 3.cs0 3:395c8678:::smithi18128752-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.5446154759999997, 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.42078353899999998, 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.691 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077158+0000", 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7540000000000001e-06 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077167+0000", 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2459999999999992e-06 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077188+0000", 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1403000000000002e-05 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077357+0000", 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000168985 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187444+0000", 2024-04-26T01:24:12.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110086218 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187592+0000", 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014808599999999999 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497585+0000", 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.309993713 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497788+0000", 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020263099999999999 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497940+0000", 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015150299999999999 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:199 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.5443717750000001, 2024-04-26T01:24:12.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41377173299999997, 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.956e-06 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077413+0000", 2024-04-26T01:24:12.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1056999999999999e-05 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077435+0000", 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2602000000000001e-05 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7252999999999999e-05 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187721+0000", 2024-04-26T01:24:12.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110268303 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187913+0000", 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000191845 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.491171+0000", 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30325871700000001 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:200 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.5442947910000004, 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41453494499999999, 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 200 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.364e-06 2024-04-26T01:24:12.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077486+0000", 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8159999999999994e-06 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077505+0000", 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9360000000000001e-05 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077521+0000", 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5685999999999999e-05 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188598+0000", 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111076802 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188834+0000", 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023592099999999999 2024-04-26T01:24:12.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.492012+0000", 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30317799600000001 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:201 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.5442314990000003, 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41550727300000001, 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:12.706 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 201 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077541+0000", 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.314e-06 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077548+0000", 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8390000000000001e-06 2024-04-26T01:24:12.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077566+0000", 2024-04-26T01:24:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7893e-05 2024-04-26T01:24:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077581+0000", 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5290000000000001e-05 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.189776+0000", 2024-04-26T01:24:12.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112194944 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.190248+0000", 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047181099999999999 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.493047+0000", 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.302799182 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.712 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:12.713 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:12.713 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:12.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:24:12.715 INFO:tasks.rados.rados.0.smithi181.stdout:100: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:24:12.716 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:12.716+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:12.815 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:12.819+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:12.842 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:12.842 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:12.842 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:12.842 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:12.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:24:12.916 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:12.920+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:12.979 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:12.979 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:12.979 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:12.979 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:12.979 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:12.986 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 4 to smithi18128752-43 2024-04-26T01:24:12.986 INFO:tasks.rados.rados.0.smithi181.stdout:99: oid 43 updating version 0 to 21 2024-04-26T01:24:12.986 INFO:tasks.rados.rados.0.smithi181.stdout:99: oid 43 version 21 is already newer than 20 2024-04-26T01:24:12.986 INFO:tasks.rados.rados.0.smithi181.stdout:99: oid 43 version 21 is already newer than 17 2024-04-26T01:24:12.986 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 21 (ObjNum 58 snap 4 seq_num 58) dirty exists 2024-04-26T01:24:12.986 INFO:tasks.rados.rados.0.smithi181.stdout:99: left oid 43 (ObjNum 58 snap 4 seq_num 58) 2024-04-26T01:24:12.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:24:13.016 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:13.020+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-04-26T01:24:13.090 INFO:tasks.rados.rados.0.smithi181.stdout:94: done (4 left) 2024-04-26T01:24:13.090 INFO:tasks.rados.rados.0.smithi181.stdout:95: done (3 left) 2024-04-26T01:24:13.090 INFO:tasks.rados.rados.0.smithi181.stdout:99: done (2 left) 2024-04-26T01:24:13.090 INFO:tasks.rados.rados.0.smithi181.stdout:100: done (1 left) 2024-04-26T01:24:13.090 INFO:tasks.rados.rados.0.smithi181.stdout:101: done (0 left) 2024-04-26T01:24:13.090 INFO:tasks.rados.rados.0.smithi181.stdout:102: snap_remove snap 1 2024-04-26T01:24:13.118 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:13.117+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s4 33/30 ECSubWrite(tid=145, reqid=client.4589.0:19, at_version=33'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:03.012957+0000", 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.111064317, 2024-04-26T01:24:13.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039332050000000004, 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.012957+0000", 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.012957+0000", 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.012958+0000", 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4200000000000001e-07 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.013136+0000", 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017799 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.013178+0000", 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1594000000000002e-05 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.013210+0000", 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2310000000000001e-05 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.013445+0000", 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000234754 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.013462+0000", 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7255e-05 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.016828+0000", 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033659969999999999 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:03.016891+0000", 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2862999999999998e-05 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 35/30 ECSubWrite(tid=15, reqid=client.4589.0:142, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.183393+0000", 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.9406285600000004, 2024-04-26T01:24:13.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102129083, 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183393+0000", 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183393+0000", 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183396+0000", 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4669999999999998e-06 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183618+0000", 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022251700000000001 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183651+0000", 2024-04-26T01:24:13.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2456000000000002e-05 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183678+0000", 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7742999999999998e-05 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.283894+0000", 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10021598 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.283913+0000", 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8895000000000001e-05 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.285439+0000", 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015254000000000001 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.285522+0000", 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3625e-05 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 35/30 ECSubWrite(tid=16, reqid=client.4589.0:143, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.286323+0000", 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.8376985149999996, 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023969600000000001, 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286323+0000", 2024-04-26T01:24:13.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286323+0000", 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286325+0000", 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1270000000000001e-06 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286472+0000", 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014652900000000001 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286503+0000", 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0836000000000001e-05 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286519+0000", 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6215999999999999e-05 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286659+0000", 2024-04-26T01:24:13.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000140123 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.286673+0000", 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3557000000000001e-05 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.288646+0000", 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001973228 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.288720+0000", 2024-04-26T01:24:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4344000000000005e-05 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.0s2 35/30 ECSubRead(tid=35, to_read={3:014d0a8d:::smithi18128752-40:head=0,4096,0}, subchunks={3:014d0a8d:::smithi18128752-40:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.374643+0000", 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.7493789179999997, 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100725563, 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.374643+0000", 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.374643+0000", 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.374645+0000", 2024-04-26T01:24:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6899999999999999e-06 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.374662+0000", 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7065999999999999e-05 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.374695+0000", 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3589e-05 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.474796+0000", 2024-04-26T01:24:13.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100101063 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.474986+0000", 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019017900000000001 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.475368+0000", 2024-04-26T01:24:13.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00038197600000000002 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/34 ECSubWrite(tid=9, reqid=client.4589.0:156, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.729293+0000", 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.3947291479999997, 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024664819999999999, 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729293+0000", 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729293+0000", 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729295+0000", 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.877e-06 2024-04-26T01:24:13.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729656+0000", 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000361307 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729689+0000", 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3073e-05 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729705+0000", 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6368999999999999e-05 2024-04-26T01:24:13.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729860+0000", 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015488899999999999 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.729890+0000", 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9995999999999999e-05 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.731695+0000", 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018050589999999999 2024-04-26T01:24:13.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.731759+0000", 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3911999999999994e-05 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/34 ECSubWrite(tid=11, reqid=client.4589.0:158, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.735791+0000", 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.3882309389999996, 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101625586, 2024-04-26T01:24:13.142 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735791+0000", 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735791+0000", 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735792+0000", 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.553e-06 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735830+0000", 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7635999999999997e-05 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735860+0000", 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9512000000000001e-05 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735875+0000", 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5353999999999999e-05 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.836096+0000", 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100221269 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.836121+0000", 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4655999999999999e-05 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.837346+0000", 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001225304 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.837416+0000", 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0302000000000006e-05 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.146 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/30 ECSubWrite(tid=172, reqid=client.4589.0:167, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.7163917360000003, 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102386068, 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.03e-07 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407939+0000", 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030856200000000002 2024-04-26T01:24:13.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407967+0000", 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7869e-05 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407983+0000", 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6404000000000001e-05 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.508232+0000", 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100248772 2024-04-26T01:24:13.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.508247+0000", 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4663e-05 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.509938+0000", 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001691084 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.510016+0000", 2024-04-26T01:24:13.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8311000000000003e-05 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/34 ECSubWrite(tid=16, reqid=client.4589.0:181, at_version=37'9, trim_to=0'0, roll_forward_to=36'8))", 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.072396+0000", 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.0516256840000002, 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047843210000000002, 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.072396+0000", 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.072396+0000", 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.072397+0000", 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5000000000000003e-07 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.072829+0000", 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00043189900000000002 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.072863+0000", 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4279000000000002e-05 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.072892+0000", 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9479e-05 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.073112+0000", 2024-04-26T01:24:13.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021956800000000001 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.073138+0000", 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6245999999999999e-05 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077054+0000", 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039155129999999998 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077180+0000", 2024-04-26T01:24:13.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126787 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/34 ECSubWrite(tid=17, reqid=client.4589.0:182, at_version=37'10, trim_to=0'0, roll_forward_to=36'8))", 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.075676+0000", 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.048345748, 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034936419999999999, 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.075676+0000", 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.075676+0000", 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.075678+0000", 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.807e-06 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.075947+0000", 2024-04-26T01:24:13.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026922300000000002 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.075978+0000", 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0540000000000002e-05 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.076000+0000", 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2422e-05 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077384+0000", 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013843250000000001 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077415+0000", 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0422000000000001e-05 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.079106+0000", 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001691131 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.079170+0000", 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3771999999999999e-05 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/34 ECSubWrite(tid=18, reqid=client.4589.0:183, at_version=37'11, trim_to=0'0, roll_forward_to=36'8))", 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077110+0000", 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.0469114639999999, 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100440694, 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.159 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077110+0000", 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077110+0000", 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077112+0000", 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7969999999999999e-06 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077157+0000", 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5148000000000002e-05 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077191+0000", 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3380000000000002e-05 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077215+0000", 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3978999999999999e-05 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.176298+0000", 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099083194999999999 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.176311+0000", 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.308e-05 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.177485+0000", 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011742440000000001 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.177551+0000", 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5870999999999993e-05 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 37/30 ECSubWrite(tid=47, reqid=client.4589.0:198, at_version=37'11, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.188295+0000", 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.935726989, 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002693006, 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.163 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188295+0000", 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188295+0000", 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188296+0000", 2024-04-26T01:24:13.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.474e-06 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188333+0000", 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6415000000000003e-05 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188365+0000", 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1949999999999997e-05 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188390+0000", 2024-04-26T01:24:13.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5446e-05 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188714+0000", 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032421499999999998 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188742+0000", 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8002999999999999e-05 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.190881+0000", 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021387820000000001 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.190988+0000", 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106721 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/34 ECSubWrite(tid=19, reqid=client.4589.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=37'11))", 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:13.167 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.8428280729999997, 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103193194, 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:13.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281196+0000", 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.046e-06 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281226+0000", 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9762999999999999e-05 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281269+0000", 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3059000000000002e-05 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281295+0000", 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6432999999999999e-05 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281433+0000", 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013830900000000001 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281449+0000", 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5846999999999999e-05 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384326+0000", 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102877183 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384387+0000", 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0554000000000003e-05 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.171 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.6s3 37/34 ECSubRead(tid=20, to_read={3:69f31b93:::smithi18128752-39:head=0,4096,0}, subchunks={3:69f31b93:::smithi18128752-39:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.282222+0000", 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.8418000890000004, 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10090128600000001, 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282222+0000", 2024-04-26T01:24:13.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282222+0000", 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282224+0000", 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0200000000000001e-06 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282246+0000", 2024-04-26T01:24:13.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1936000000000001e-05 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282285+0000", 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9431999999999998e-05 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282317+0000", 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1622999999999999e-05 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.382647+0000", 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100330423 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.383123+0000", 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047585200000000002 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.0s2 37/30 ECSubRead(tid=48, to_read={3:0c3f65fa:::smithi18128752-8:head=0,32768,0}, subchunks={3:0c3f65fa:::smithi18128752-8:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.286865+0000", 2024-04-26T01:24:13.175 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.8371570999999998, 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100797022, 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.286865+0000", 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.286865+0000", 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.286866+0000", 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7269999999999999e-06 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.286885+0000", 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8649000000000001e-05 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.286922+0000", 2024-04-26T01:24:13.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6766e-05 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.286950+0000", 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7685999999999999e-05 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.387158+0000", 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10020894 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.387662+0000", 2024-04-26T01:24:13.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00050325399999999996 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 41/34 ECSubWrite(tid=27, reqid=osd.0.0:27, at_version=41'17, trim_to=0'0, roll_forward_to=39'15))", 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.099781+0000", 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.0242403689999999, 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034444330000000002, 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.099781+0000", 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.099781+0000", 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.099782+0000", 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1100000000000002e-07 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.099804+0000", 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.177e-05 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.099957+0000", 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015396600000000001 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.099971+0000", 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4008999999999999e-05 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.100084+0000", 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011264600000000001 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.100099+0000", 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5138e-05 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.103185+0000", 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030860810000000001 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.103226+0000", 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0512000000000001e-05 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 41/30 ECSubWrite(tid=32, reqid=client.4589.0:266, at_version=41'17, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.175266+0000", 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.94875594200000002, 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035117600000000001, 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.183 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175266+0000", 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175266+0000", 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175266+0000", 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6e-07 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175674+0000", 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040823999999999998 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175691+0000", 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6884e-05 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175704+0000", 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2449000000000001e-05 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175874+0000", 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017013299999999999 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.175906+0000", 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2391999999999998e-05 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.178730+0000", 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028239049999999998 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.178778+0000", 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7596999999999998e-05 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s4 41/30 ECSubWrite(tid=178, reqid=osd.13.0:178, at_version=41'13, trim_to=0'0, roll_forward_to=39'11))", 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.199973+0000", 2024-04-26T01:24:13.187 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.92404898800000002, 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035326839999999999, 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.199973+0000", 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.199973+0000", 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.199973+0000", 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.34e-07 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.200002+0000", 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8605000000000001e-05 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.200019+0000", 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.679e-05 2024-04-26T01:24:13.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.200031+0000", 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2112e-05 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.202698+0000", 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002666987 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.202708+0000", 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0733e-05 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.203433+0000", 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000725065 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.203505+0000", 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2057999999999993e-05 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 41/30 ECSubWrite(tid=35, reqid=client.4589.0:268, at_version=41'21, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.283921+0000", 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.84010078399999999, 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101082875, 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283921+0000", 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283921+0000", 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283924+0000", 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6539999999999999e-06 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283969+0000", 2024-04-26T01:24:13.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5082e-05 2024-04-26T01:24:13.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.284005+0000", 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5836e-05 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.284024+0000", 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9760000000000001e-05 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.384220+0000", 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100195435 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.384228+0000", 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5080000000000005e-06 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.384974+0000", 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000745716 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.385004+0000", 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9884000000000001e-05 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 41/34 ECSubWrite(tid=28, reqid=osd.0.0:27, at_version=0'0, trim_to=0'0, roll_forward_to=41'17))", 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.412602+0000", 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.71142014499999995, 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10092269800000001, 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412602+0000", 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412602+0000", 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412603+0000", 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.392e-06 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412616+0000", 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2778e-05 2024-04-26T01:24:13.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412631+0000", 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4938e-05 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.512740+0000", 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100108817 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.512822+0000", 2024-04-26T01:24:13.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2826999999999998e-05 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.512836+0000", 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.364e-05 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.513487+0000", 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065126699999999997 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.513524+0000", 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7039000000000002e-05 2024-04-26T01:24:13.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.62802726600000003, 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10316732200000001, 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:13.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495996+0000", 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.457e-06 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496024+0000", 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8269999999999999e-05 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496047+0000", 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3045e-05 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496077+0000", 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0046999999999999e-05 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.596334+0000", 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10025700899999999 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.596353+0000", 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8423e-05 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.599074+0000", 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002720943 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:13.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.599162+0000", 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8127999999999994e-05 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:13.205 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:13.206 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:24:13.219 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:13.220+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:13.319 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:13.323+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:13.342 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:13.342 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:13.342 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:13.342 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:13.357 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:24:13.420 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:13.424+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:13.521 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:13.520+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:13.527 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:13.527 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:13.527 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:13.527 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:13.527 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:13.542 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:24:13.622 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:13.625+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:13.664 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.664 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:115 3.6s0 3:69f31b93:::smithi18128752-39:head [append 0~638976,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 8.2799958510000007, 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.27460739099999998, 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 115 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385705+0000", 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5990000000000001e-06 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386759+0000", 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010545940000000001 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386792+0000", 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2376999999999998e-05 2024-04-26T01:24:13.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386816+0000", 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4680000000000001e-05 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.454087+0000", 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.067270658999999997 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.454285+0000", 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019791700000000001 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.455558+0000", 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001272731 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558702+0000", 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103144082 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:13.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.659941+0000", 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101238869 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.660311+0000", 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000369883 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:116 3.6s0 3:69f31b93:::smithi18128752-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:13.670 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 8.2788441049999992, 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.37357116200000001, 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 116 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:13.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386857+0000", 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.703e-06 2024-04-26T01:24:13.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386866+0000", 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5670000000000001e-06 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386887+0000", 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1316000000000001e-05 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386904+0000", 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6767000000000001e-05 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.556964+0000", 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17005975700000001 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.557176+0000", 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021186600000000001 2024-04-26T01:24:13.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558110+0000", 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00093395700000000002 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.559561+0000", 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014508430000000001 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.759894+0000", 2024-04-26T01:24:13.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033284600000001 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.760426+0000", 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00053253999999999997 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:117 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 8.2787694520000006, 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77605625899999997, 2024-04-26T01:24:13.676 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 117 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386931+0000", 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3599999999999999e-06 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386939+0000", 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8369999999999997e-06 2024-04-26T01:24:13.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386957+0000", 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8683000000000001e-05 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386974+0000", 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558747+0000", 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17177335099999999 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558933+0000", 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018625099999999999 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761103+0000", 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202169231 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761231+0000", 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000128734 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162986+0000", 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.401754422 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 35/30 ECSubWrite(tid=12, reqid=client.4589.0:128, at_version=35'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.580804+0000", 2024-04-26T01:24:13.681 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 8.0848950580000007, 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20196134800000001, 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580804+0000", 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580804+0000", 2024-04-26T01:24:13.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580805+0000", 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.494e-06 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580833+0000", 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7373e-05 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580858+0000", 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4819000000000001e-05 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.680966+0000", 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100108062 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.781188+0000", 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100222797 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.781217+0000", 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8563000000000001e-05 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.782727+0000", 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001509727 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.782765+0000", 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8513000000000002e-05 2024-04-26T01:24:13.685 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 35/34 ECSubWrite(tid=9, reqid=client.4589.0:71, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.681169+0000", 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.9845304339999998, 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200637708, 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.681169+0000", 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.681169+0000", 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.681169+0000", 2024-04-26T01:24:13.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0999999999999999e-07 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.681195+0000", 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5933000000000001e-05 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.681208+0000", 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2996999999999999e-05 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.781295+0000", 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100086812 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.781600+0000", 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030561299999999999 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.781623+0000", 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2486000000000001e-05 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.881770+0000", 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10014706700000001 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.881806+0000", 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6390000000000002e-05 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-04-26T01:24:13.690 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.862014+0000", 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.8036846820000001, 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200363084, 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862014+0000", 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862014+0000", 2024-04-26T01:24:13.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862015+0000", 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3500000000000005e-07 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862021+0000", 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9529999999999996e-06 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862033+0000", 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.169e-05 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862053+0000", 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0520999999999999e-05 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.062355+0000", 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20030187299999999 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.062377+0000", 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2212e-05 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.862299+0000", 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.8034003800000002, 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20016621200000001, 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.694 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862299+0000", 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862299+0000", 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862300+0000", 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8199999999999998e-07 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862308+0000", 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2209999999999997e-06 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.862321+0000", 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3565e-05 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962405+0000", 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10008414 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.062435+0000", 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100029862 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.062465+0000", 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9541999999999999e-05 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.697 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.cs2 35/34 ECSubRead(tid=12, to_read={3:3c2c778f:::smithi18128752-24:head=0,4096,0}, subchunks={3:3c2c778f:::smithi18128752-24:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.953255+0000", 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.712444015, 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200861238, 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953255+0000", 2024-04-26T01:24:13.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953255+0000", 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953257+0000", 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.26e-06 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953279+0000", 2024-04-26T01:24:13.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1943999999999999e-05 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.953322+0000", 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2632000000000001e-05 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.053433+0000", 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100111437 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.153674+0000", 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100240262 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.154116+0000", 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00044270300000000002 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=6, last_complete=35'4, committed=1, applied=1))", 2024-04-26T01:24:13.701 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.962548+0000", 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.7031511310000003, 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20001797499999999, 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962548+0000", 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962548+0000", 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962548+0000", 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2500000000000001e-07 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962552+0000", 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3459999999999998e-06 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962560+0000", 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0320000000000003e-06 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962564+0000", 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2440000000000004e-06 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162516+0000", 2024-04-26T01:24:13.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19995191700000001 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162566+0000", 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0010999999999999e-05 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 35/34 ECSubReadReply(tid=7, attrs_read=0))", 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.962571+0000", 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.7031276220000002, 2024-04-26T01:24:13.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20042953199999999, 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962571+0000", 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962571+0000", 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962572+0000", 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.98e-07 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962579+0000", 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5039999999999996e-06 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962586+0000", 2024-04-26T01:24:13.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4829999999999996e-06 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.962589+0000", 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7840000000000001e-06 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162594+0000", 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200004451 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.163001+0000", 2024-04-26T01:24:13.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000407112 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 35/30 ECSubWrite(tid=15, reqid=client.4589.0:142, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.216520+0000", 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.4491793739999999, 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20095770800000001, 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216520+0000", 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216520+0000", 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216521+0000", 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6750000000000001e-06 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216661+0000", 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014010699999999999 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216682+0000", 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0182000000000001e-05 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.316787+0000", 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10010555 2024-04-26T01:24:13.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.316920+0000", 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013260799999999999 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.316933+0000", 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3621999999999999e-05 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.417433+0000", 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10049991899999999 2024-04-26T01:24:13.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.417477+0000", 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4045000000000003e-05 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:156 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~475136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.1680637090000001, 2024-04-26T01:24:13.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33900037700000002, 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 156 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497638+0000", 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2110000000000001e-06 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498467+0000", 2024-04-26T01:24:13.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082923399999999998 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498486+0000", 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9341e-05 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498496+0000", 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8179999999999999e-06 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498610+0000", 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000114259 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498770+0000", 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016017599999999999 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726262+0000", 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22749161900000001 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726429+0000", 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000166825 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.728061+0000", 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016325490000000001 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735773+0000", 2024-04-26T01:24:13.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0077114189999999997 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836407+0000", 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10063380399999999 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836636+0000", 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022912200000000001 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:157 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.1618432429999999, 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.43381848000000001, 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 157 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.720 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503857+0000", 2024-04-26T01:24:13.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2920000000000001e-06 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504200+0000", 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034286599999999999 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504218+0000", 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7961e-05 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504227+0000", 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6240000000000008e-06 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504381+0000", 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015489900000000001 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504555+0000", 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017342900000000001 2024-04-26T01:24:13.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.729729+0000", 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225174034 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.730038+0000", 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030876200000000002 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.732244+0000", 2024-04-26T01:24:13.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022065190000000001 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735776+0000", 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035319330000000001 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937311+0000", 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201534612 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937674+0000", 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00036354900000000001 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:158 3.6s0 3:60f0095a:::smithi18128752-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.1614521709999996, 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.630196439, 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.726 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 158 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504255+0000", 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0989999999999999e-06 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504262+0000", 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5660000000000003e-06 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504266+0000", 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0980000000000004e-06 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504650+0000", 2024-04-26T01:24:13.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00038378400000000002 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504796+0000", 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014592199999999999 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733685+0000", 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22888902699999999 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733975+0000", 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029027399999999998 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.734990+0000", 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010148500000000001 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836708+0000", 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10171771 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134326+0000", 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29761846199999997 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134443+0000", 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117142 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.732 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:159 3.6s0 3:60f0095a:::smithi18128752-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.1614265460000004, 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.73307043299999997, 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 159 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504273+0000", 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3500000000000002e-07 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504274+0000", 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.562e-06 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504281+0000", 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3790000000000003e-06 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504286+0000", 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1989999999999997e-06 2024-04-26T01:24:13.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504861+0000", 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00057502400000000004 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504981+0000", 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012016899999999999 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837238+0000", 2024-04-26T01:24:13.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33225718900000001 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837352+0000", 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113583 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134907+0000", 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29755462599999999 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.135101+0000", 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019402500000000001 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.237343+0000", 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102242342 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.738 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=9, last_complete=35'5, committed=1, applied=1))", 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.731425+0000", 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.9342735189999996, 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10475909, 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.731425+0000", 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.731425+0000", 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.731427+0000", 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3069999999999999e-06 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.731434+0000", 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1729999999999998e-06 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.731455+0000", 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0733999999999998e-05 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.731466+0000", 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1e-05 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836151+0000", 2024-04-26T01:24:13.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10468559700000001 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836185+0000", 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3278999999999998e-05 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=11, last_complete=35'7, committed=1, applied=1))", 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.83155248, 2024-04-26T01:24:13.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30069599499999999, 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0800000000000003e-07 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834153+0000", 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1699999999999996e-06 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834168+0000", 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5017999999999999e-05 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.934244+0000", 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100075936 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134251+0000", 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200007516 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134843+0000", 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059144700000000004 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:171 3.6s0 3:69f31b93:::smithi18128752-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.3290036089999999, 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36489432300000002, 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.746 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 171 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336697+0000", 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3829999999999999e-06 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336701+0000", 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.053e-06 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336710+0000", 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8349999999999993e-06 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336715+0000", 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5219999999999998e-06 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337072+0000", 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00035695799999999999 2024-04-26T01:24:13.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337245+0000", 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017283899999999999 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.338075+0000", 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082985399999999999 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.339323+0000", 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012477390000000001 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701394+0000", 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36207125499999998 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701590+0000", 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000195885 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds4 41/30 ECSubWrite(tid=93, reqid=osd.4.0:93, at_version=41'16, trim_to=0'0, roll_forward_to=39'14))", 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.405660+0000", 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.260039154, 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103230932, 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.752 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.405660+0000", 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.405660+0000", 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.405661+0000", 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.637e-06 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.405710+0000", 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8785999999999997e-05 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.405742+0000", 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1587000000000001e-05 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.505892+0000", 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100150525 2024-04-26T01:24:13.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506050+0000", 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000157229 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506074+0000", 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4213999999999999e-05 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.508823+0000", 2024-04-26T01:24:13.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002748688 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.508891+0000", 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8266e-05 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s3 41/30 ECSubWrite(tid=97, reqid=osd.4.0:97, at_version=41'22, trim_to=0'0, roll_forward_to=39'18))", 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.159198969, 2024-04-26T01:24:13.756 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201986786, 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:13.757 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506501+0000", 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0100000000000001e-06 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506544+0000", 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2605999999999998e-05 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506567+0000", 2024-04-26T01:24:13.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3269999999999999e-05 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.606669+0000", 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10010197899999999 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.706960+0000", 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10029147400000001 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.706979+0000", 2024-04-26T01:24:13.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8403e-05 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.708374+0000", 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013951510000000001 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.708487+0000", 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000112893 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:13.760 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:13.761 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:13.761 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:13.762 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:13.768+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:13.763 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:24:13.863 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:13.868+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:13.932 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:13.932 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:13.932 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:13.932 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:13.949 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:24:13.961 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:13.961 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:13.964 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:13.964+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:13.982 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:13.982 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:14.003 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:14.003 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:14.024 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:14.024 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:14.042 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:14.042 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:14.063 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:14.063 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:14.064 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:14.064 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:14.065 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:14.064+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:14.079 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:24:14.085 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:14.085 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:102: done (0 left) 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:103: read oid 29 snap 4 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:103: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:104: read oid 46 snap -1 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:104: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:105: copy_from oid 36 from oid 2 current snap is 5 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:106: rmattr oid 49 current snap is 5 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:106: done (3 left) 2024-04-26T01:24:14.107 INFO:tasks.rados.rados.0.smithi181.stdout:107: snap_remove snap 3 2024-04-26T01:24:14.114 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:14.114 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:14.166 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:14.167+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:14.189 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15] out_osds: [7, 12] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:24:14.189 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:24:14.189 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15] 2024-04-26T01:24:14.189 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-04-26T01:24:14.209 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:14.209 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:14.226 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:24:14.228 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:14.229 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:14.250 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:14.250 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:14.267 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:14.271+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:14.271 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:14.272 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:14.297 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:14.298 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:14.323 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:14.324 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:14.344 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:14.344 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:14.359 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:14.359 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:14.364 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:14.365 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:14.368 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:14.371+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:14.374 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:24:14.436 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:14.436 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:14.503 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:14.507+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:14.512 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T01:24:14.515+0000 7f533fe62640 -1 mon.a@0(leader).osd e43 definitely_dead 0 2024-04-26T01:24:14.532 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:14.532 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:14.547 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:24:14.591 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:14.595+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:14.671 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:14.676+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:14.719 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:14.719 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:14.733 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:24:14.779 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:14.783+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:14.856 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:14.856 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:14.872 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:24:14.873 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:14.876+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:14.974 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:14.976+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:15.045 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:15.045 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:15.060 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:24:15.108 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:24:15.109 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:15.111+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:15.110 INFO:teuthology.orchestra.run.smithi037.stderr:marked out osd.2. 2024-04-26T01:24:15.117 INFO:tasks.rados.rados.0.smithi181.stdout:107: done (3 left) 2024-04-26T01:24:15.117 INFO:tasks.rados.rados.0.smithi181.stdout:108: rmattr oid 30 current snap is 5 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:108: done (3 left) 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:109: read oid 42 snap 2 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:109: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:110: rmattr oid 14 current snap is 5 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:110: done (4 left) 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:111: read oid 9 snap 2 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:111: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 24 current snap is 5 2024-04-26T01:24:15.118 INFO:tasks.rados.rados.0.smithi181.stdout:104: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:15.177 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:15.180+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:15.185 INFO:tasks.rados.rados.0.smithi181.stdout:105: finishing copy_from racing read to smithi18128752-36 2024-04-26T01:24:15.185 INFO:tasks.rados.rados.0.smithi181.stdout:105: finishing copy_from to smithi18128752-36 2024-04-26T01:24:15.185 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-04-26T01:24:15.185 INFO:tasks.rados.rados.0.smithi181.stdout:103: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:24:15.197 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:15.197 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:15.204 INFO:tasks.rados.rados.0.smithi181.stdout:112: seq_num 59 ranges {393216=622592,1015808=376832} 2024-04-26T01:24:15.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:24:15.221 INFO:tasks.rados.rados.0.smithi181.stdout:112: writing smithi18128752-24 from 393216 to 1015808 tid 1 2024-04-26T01:24:15.229 INFO:tasks.rados.rados.0.smithi181.stdout:112: writing smithi18128752-24 from 1015808 to 1392640 tid 2 2024-04-26T01:24:15.229 INFO:tasks.rados.rados.0.smithi181.stdout:103: done (5 left) 2024-04-26T01:24:15.229 INFO:tasks.rados.rados.0.smithi181.stdout:104: done (4 left) 2024-04-26T01:24:15.229 INFO:tasks.rados.rados.0.smithi181.stdout:105: done (3 left) 2024-04-26T01:24:15.229 INFO:tasks.rados.rados.0.smithi181.stdout:113: copy_from oid 34 from oid 40 current snap is 5 2024-04-26T01:24:15.229 INFO:tasks.rados.rados.0.smithi181.stdout:114: snap_remove snap 4 2024-04-26T01:24:15.278 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:15.279+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:15.348 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:15.348 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:15.363 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:24:15.410 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:15.412+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:15.479 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:15.480+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:15.488 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:15.488 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:15.502 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:24:15.590 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:15.592+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:15.674 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:15.675 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:15.680 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:15.683+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:15.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:24:15.781 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:15.780+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:15.862 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:15.862 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:15.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:24:15.881 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:15.884+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:15.982 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:15.984+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:16.048 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:16.048 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:16.063 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:24:16.083 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:16.084+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:16.121 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:24:16.128 INFO:tasks.rados.rados.0.smithi181.stdout:114: done (4 left) 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:115: copy_from oid 25 from oid 30 current snap is 5 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: finishing write tid 1 to smithi18128752-24 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: finishing write tid 2 to smithi18128752-24 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: finishing write tid 3 to smithi18128752-24 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: finishing write tid 4 to smithi18128752-24 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: oid 24 updating version 0 to 17 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: oid 24 version 17 is already newer than 16 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: oid 24 version 17 is already newer than 15 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 17 (ObjNum 59 snap 5 seq_num 59) dirty exists 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: left oid 24 (ObjNum 59 snap 5 seq_num 59) 2024-04-26T01:24:16.129 INFO:tasks.rados.rados.0.smithi181.stdout:112: done (4 left) 2024-04-26T01:24:16.130 INFO:tasks.rados.rados.0.smithi181.stdout:116: read oid 10 snap 2 2024-04-26T01:24:16.130 INFO:tasks.rados.rados.0.smithi181.stdout:116: expect (ObjNum 9 snap 0 seq_num 9) 2024-04-26T01:24:16.130 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 37 current snap is 5 2024-04-26T01:24:16.130 INFO:tasks.rados.rados.0.smithi181.stdout:117: seq_num 60 ranges {606208=638976,1245184=376832} 2024-04-26T01:24:16.151 INFO:tasks.rados.rados.0.smithi181.stdout:117: writing smithi18128752-37 from 606208 to 1245184 tid 1 2024-04-26T01:24:16.156 INFO:tasks.rados.rados.0.smithi181.stdout:117: writing smithi18128752-37 from 1245184 to 1622016 tid 2 2024-04-26T01:24:16.157 INFO:tasks.rados.rados.0.smithi181.stdout:118: delete oid 23 current snap is 5 2024-04-26T01:24:16.184 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:16.184+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:16.243 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:16.243 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:16.257 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:24:16.285 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:16.285+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:16.386 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:16.388+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:16.429 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:16.429 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:16.447 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:16.462 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:16.490 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:16.492+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:16.587 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:16.591+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:16.612 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:16.612 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:16.621 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:16.621 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:16.627 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:16.637 INFO:tasks.rados.rados.0.smithi181.stdout:116: expect (ObjNum 9 snap 0 seq_num 9) 2024-04-26T01:24:16.642 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:16.642 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:16.663 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:16.664 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:16.684 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:16.685 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:16.688 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:16.691+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:16.701 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:16.701 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:16.720 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:16.720 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:16.742 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:16.742 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:16.747 INFO:tasks.rados.rados.0.smithi181.stdout:115: finishing copy_from racing read to smithi18128752-25 2024-04-26T01:24:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:16.751 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:16.765 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:16.765 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:16.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:16.783 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:16.783 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:16.789 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:16.792+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:16.803 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:16.803 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:16.824 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:16.825 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:16.847 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:16.847 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:16.867 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:16.867 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:16.889 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:16.890 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:16.890 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:16.889+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:16.895 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:16.895 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:16.895 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-04-26T01:24:16.895 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:16.909 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:16.912 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:16.912 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:16.937 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:16.938 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:16.991 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:16.989+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:17.030 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:17.030 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:17.030 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:17.030 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:17.030 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:17.041 INFO:tasks.rados.rados.0.smithi181.stdout:115: finishing copy_from to smithi18128752-25 2024-04-26T01:24:17.041 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 25 v 10 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:24:17.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:17.048 INFO:tasks.rados.rados.0.smithi181.stdout:117: finishing write tid 1 to smithi18128752-37 2024-04-26T01:24:17.061 INFO:tasks.rados.rados.0.smithi181.stdout:109: expect (ObjNum 41 snap 0 seq_num 41) 2024-04-26T01:24:17.092 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:17.095+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:17.176 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:24:17.176 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.548452028, 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40356932699999998, 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 37 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.177 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4700000000000001e-07 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.178 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630292+0000", 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3669999999999999e-06 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630302+0000", 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0115999999999999e-05 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630310+0000", 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2029999999999992e-06 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.179 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735212+0000", 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104902254 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735342+0000", 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012980500000000001 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735542+0000", 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020034 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.180 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.736506+0000", 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00096400499999999998 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033692+0000", 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29718516499999997 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033783+0000", 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1377000000000003e-05 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.181 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033853+0000", 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9948000000000002e-05 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.247345743, 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40787895499999999, 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.182 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 84 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.183 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931392+0000", 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7659999999999999e-06 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931546+0000", 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015429199999999999 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931571+0000", 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5381999999999999e-05 2024-04-26T01:24:17.184 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931597+0000", 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5616e-05 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934003+0000", 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024058180000000001 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934128+0000", 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505200000000001 2024-04-26T01:24:17.185 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934351+0000", 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000222912 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935791+0000", 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014398799999999999 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339098+0000", 2024-04-26T01:24:17.186 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40330716900000002 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339195+0000", 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7676999999999993e-05 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339269+0000", 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3391000000000003e-05 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.187 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.247103538000001, 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.407895061, 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 85 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931634+0000", 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4330000000000001e-06 2024-04-26T01:24:17.189 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931646+0000", 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2622999999999999e-05 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931665+0000", 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8669999999999999e-05 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931699+0000", 2024-04-26T01:24:17.190 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4112000000000001e-05 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934956+0000", 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032571119999999999 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935079+0000", 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122483 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935269+0000", 2024-04-26T01:24:17.191 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019042700000000001 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.936573+0000", 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013043340000000001 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339389+0000", 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.402815335 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339468+0000", 2024-04-26T01:24:17.192 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9758000000000004e-05 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339527+0000", 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8773999999999997e-05 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.247008544, 2024-04-26T01:24:17.193 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40806487499999999, 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 86 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.194 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931728+0000", 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3030000000000001e-06 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931739+0000", 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.046e-05 2024-04-26T01:24:17.195 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931758+0000", 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8842e-05 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.031855+0000", 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100097588 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032039+0000", 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000184141 2024-04-26T01:24:17.196 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032212+0000", 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000172747 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032428+0000", 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021604 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.135784+0000", 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10335620700000001 2024-04-26T01:24:17.197 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339636+0000", 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20385123199999999 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339735+0000", 2024-04-26T01:24:17.198 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9036999999999994e-05 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339792+0000", 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7278000000000003e-05 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.146636612000002, 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40781067500000001, 2024-04-26T01:24:17.199 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.200 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032101+0000", 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7630000000000001e-06 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032115+0000", 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4375999999999999e-05 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.201 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032137+0000", 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1356e-05 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032158+0000", 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1253000000000001e-05 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134605+0000", 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102446813 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.202 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134733+0000", 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012879999999999999 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134935+0000", 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020153599999999999 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.136395+0000", 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014603089999999999 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.203 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439763+0000", 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30336767999999997 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439846+0000", 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3430999999999995e-05 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439910+0000", 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3357999999999999e-05 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.204 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405207+0000", 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.773528827, 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.308960927, 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 107 2024-04-26T01:24:17.205 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405207+0000", 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405207+0000", 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.206 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405208+0000", 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8299999999999995e-07 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405214+0000", 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6000000000000003e-06 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405225+0000", 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1005999999999999e-05 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405233+0000", 2024-04-26T01:24:17.207 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1359999999999997e-06 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.409297+0000", 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040637110000000002 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.409532+0000", 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000234742 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.409912+0000", 2024-04-26T01:24:17.208 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000380178 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414844+0000", 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0049318599999999997 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.713884+0000", 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29904050700000001 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.209 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714051+0000", 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166754 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714168+0000", 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011655 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405271+0000", 2024-04-26T01:24:17.210 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.773464495999999, 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30920326599999998, 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 108 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405271+0000", 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.211 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405271+0000", 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405272+0000", 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405279+0000", 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6599999999999998e-06 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.212 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405291+0000", 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2211999999999999e-05 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405300+0000", 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7669999999999992e-06 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.411128+0000", 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0058281779999999998 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.213 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.411372+0000", 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024450900000000001 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.411759+0000", 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00038703700000000001 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417160+0000", 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.005400658 2024-04-26T01:24:17.214 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714317+0000", 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29715656299999998 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714407+0000", 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0061000000000004e-05 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.714474+0000", 2024-04-26T01:24:17.215 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7654e-05 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405315+0000", 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.773420136999999, 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30982507199999998, 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.216 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 109 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405315+0000", 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405315+0000", 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.217 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405316+0000", 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405436+0000", 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011977999999999999 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405450+0000", 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3706000000000001e-05 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.218 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405487+0000", 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6869000000000002e-05 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.412356+0000", 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0068690410000000002 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.412602+0000", 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024654199999999999 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.219 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.413004+0000", 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00040204100000000001 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417187+0000", 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0041823590000000001 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.715000+0000", 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.2978131 2024-04-26T01:24:17.220 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.715080+0000", 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9993999999999993e-05 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.715141+0000", 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0863999999999999e-05 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.221 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.773228570000001, 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41242393999999999, 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 110 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.222 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405508+0000", 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3699999999999999e-07 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.223 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405515+0000", 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.669e-06 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405525+0000", 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0657e-05 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405533+0000", 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0160000000000008e-06 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.224 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414083+0000", 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0085497569999999998 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414207+0000", 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012396999999999999 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414400+0000", 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019254 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.225 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520650+0000", 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10625069299999999 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817668+0000", 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29701819000000002 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817820+0000", 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000151075 2024-04-26T01:24:17.226 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817931+0000", 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011143599999999999 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.773189885000001, 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51275573399999996, 2024-04-26T01:24:17.227 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 111 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.228 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6700000000000003e-07 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405552+0000", 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0930000000000002e-06 2024-04-26T01:24:17.229 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405561+0000", 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5030000000000003e-06 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405572+0000", 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0723000000000001e-05 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414943+0000", 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093715910000000003 2024-04-26T01:24:17.230 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.415187+0000", 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024357900000000001 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.415560+0000", 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00037288799999999998 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520666+0000", 2024-04-26T01:24:17.231 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105105713 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918150+0000", 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.39748471200000002 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918236+0000", 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5780999999999997e-05 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918301+0000", 2024-04-26T01:24:17.232 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5483999999999998e-05 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.773151645999999, 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51318251699999995, 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.233 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 112 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.234 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405585+0000", 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5300000000000003e-07 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405743+0000", 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015815499999999999 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405757+0000", 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3997e-05 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.235 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405767+0000", 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6260000000000007e-06 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416357+0000", 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010590268999999999 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416478+0000", 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012163300000000001 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.236 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416669+0000", 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019071999999999999 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522454+0000", 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105784641 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918478+0000", 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.396024563 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.237 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918641+0000", 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016244100000000001 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918767+0000", 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012571899999999999 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:17.238 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.772951081999999, 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61300901299999999, 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 113 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:17.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405786+0000", 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5600000000000004e-07 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405803+0000", 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7666e-05 2024-04-26T01:24:17.240 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405814+0000", 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0747000000000001e-05 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405822+0000", 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8329999999999994e-06 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417320+0000", 2024-04-26T01:24:17.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011497772999999999 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417564+0000", 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000244842 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.418017+0000", 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045273500000000001 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522469+0000", 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104451424 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018558+0000", 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.496089267 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018655+0000", 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7194000000000005e-05 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018794+0000", 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013857600000000001 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.772900850999999, 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61322511700000004, 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.244 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 114 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1699999999999997e-07 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405842+0000", 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8530000000000003e-06 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405852+0000", 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6830000000000003e-06 2024-04-26T01:24:17.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405860+0000", 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6159999999999996e-06 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.518995+0000", 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11313486 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519240+0000", 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000245593 2024-04-26T01:24:17.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519639+0000", 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039928400000000002 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524261+0000", 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046215249999999996 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018869+0000", 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49460855799999998 2024-04-26T01:24:17.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018947+0000", 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7742000000000002e-05 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019060+0000", 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112686 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.249 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.772860068, 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61349101800000005, 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.16e-07 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406016+0000", 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013951499999999999 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406029+0000", 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2819999999999999e-05 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406037+0000", 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6470000000000004e-06 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520774+0000", 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.114736465 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520941+0000", 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016684800000000001 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.521147+0000", 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020602000000000001 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524283+0000", 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003136453 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019192+0000", 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.494908399 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019283+0000", 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1893999999999998e-05 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019367+0000", 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3140999999999996e-05 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.772680088000001, 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61359576599999999, 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.255 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6400000000000001e-07 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406062+0000", 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5160000000000002e-06 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406072+0000", 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.91e-06 2024-04-26T01:24:17.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406080+0000", 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7030000000000006e-06 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522858+0000", 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116778593 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523093+0000", 2024-04-26T01:24:17.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000234763 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523479+0000", 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000385831 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525176+0000", 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016968619999999999 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019468+0000", 2024-04-26T01:24:17.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49429284899999998 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019546+0000", 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7553999999999996e-05 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019651+0000", 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000105321 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 20.772644161999999, 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61381219799999998, 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 117 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406092+0000", 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6700000000000003e-07 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406100+0000", 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4560000000000003e-06 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406109+0000", 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4590000000000004e-06 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.506278+0000", 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10016914 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524347+0000", 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018068623999999998 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524495+0000", 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148276 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524702+0000", 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020681900000000001 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525680+0000", 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00097787599999999992 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019759+0000", 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49407962300000002 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019840+0000", 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0563999999999996e-05 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019904+0000", 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3694000000000004e-05 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:187 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:00.690499+0000", 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.48823651, 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30912161999999999, 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.266 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 187 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690499+0000", 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690499+0000", 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690500+0000", 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2899999999999999e-07 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690505+0000", 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5269999999999998e-06 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690512+0000", 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8940000000000004e-06 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690518+0000", 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8209999999999998e-06 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.694555+0000", 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0040370120000000004 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.694625+0000", 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0574999999999994e-05 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.694722+0000", 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6947000000000005e-05 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.790575+0000", 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095852563000000002 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.999360+0000", 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20878496999999999 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.999498+0000", 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000138166 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.999621+0000", 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012271600000000001 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.271 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:188 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:00.690529+0000", 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.488206902999998, 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30951363100000001, 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 188 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.272 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690529+0000", 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690529+0000", 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690529+0000", 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.51e-07 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690602+0000", 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2306999999999997e-05 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690613+0000", 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0976e-05 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690620+0000", 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9670000000000004e-06 2024-04-26T01:24:17.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.695033+0000", 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044132499999999996 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.695090+0000", 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7157999999999997e-05 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.695184+0000", 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3931999999999994e-05 2024-04-26T01:24:17.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.790581+0000", 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095397412000000001 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.999758+0000", 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20917635900000001 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.999922+0000", 2024-04-26T01:24:17.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016469600000000001 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:01.000042+0000", 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000120023 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:00.690634+0000", 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.488101678, 2024-04-26T01:24:17.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30989272800000001, 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 189 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690634+0000", 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690634+0000", 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690635+0000", 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4199999999999995e-07 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690640+0000", 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2140000000000004e-06 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690648+0000", 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1040000000000007e-06 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690658+0000", 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7389999999999994e-06 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.789541+0000", 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098883440000000003 2024-04-26T01:24:17.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.789695+0000", 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000153638 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.789927+0000", 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023231400000000001 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.791113+0000", 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011859850000000001 2024-04-26T01:24:17.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:01.000384+0000", 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20927051599999999 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:01.000464+0000", 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0317999999999999e-05 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:01.000527+0000", 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2817999999999994e-05 2024-04-26T01:24:17.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:167 3.2s0 3:42c5d4a2:::smithi18128752-44:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.297589+0000", 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.8811468169999994, 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40316530499999997, 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:17.283 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 167 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297589+0000", 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297589+0000", 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297592+0000", 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.768e-06 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297606+0000", 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4229999999999999e-05 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297639+0000", 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3188000000000001e-05 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297655+0000", 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6090999999999999e-05 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297844+0000", 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018922599999999999 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.298022+0000", 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017731600000000001 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.406673+0000", 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.108651228 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.408960+0000", 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002286883 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.700582+0000", 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29162229299999998 2024-04-26T01:24:17.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.700754+0000", 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000172082 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:17.288 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:17.290 INFO:tasks.rados.rados.0.smithi181.stdout:113: finishing copy_from racing read to smithi18128752-34 2024-04-26T01:24:17.290 INFO:tasks.rados.rados.0.smithi181.stdout:117: finishing write tid 2 to smithi18128752-37 2024-04-26T01:24:17.290 INFO:tasks.rados.rados.0.smithi181.stdout:117: finishing write tid 3 to smithi18128752-37 2024-04-26T01:24:17.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:17.291 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:17.292+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:17.391 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:17.395+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:17.420 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:17.420 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:17.420 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:17.420 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:17.434 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:17.438 INFO:tasks.rados.rados.0.smithi181.stdout:113: finishing copy_from to smithi18128752-34 2024-04-26T01:24:17.439 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T01:24:17.492 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:17.496+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:17.553 INFO:tasks.rados.rados.0.smithi181.stdout:117: finishing write tid 4 to smithi18128752-37 2024-04-26T01:24:17.554 INFO:tasks.rados.rados.0.smithi181.stdout:117: oid 37 updating version 0 to 12 2024-04-26T01:24:17.554 INFO:tasks.rados.rados.0.smithi181.stdout:117: oid 37 updating version 12 to 13 2024-04-26T01:24:17.554 INFO:tasks.rados.rados.0.smithi181.stdout:117: oid 37 updating version 13 to 14 2024-04-26T01:24:17.554 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 14 (ObjNum 60 snap 5 seq_num 60) dirty exists 2024-04-26T01:24:17.554 INFO:tasks.rados.rados.0.smithi181.stdout:117: left oid 37 (ObjNum 60 snap 5 seq_num 60) 2024-04-26T01:24:17.556 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:17.556 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:17.556 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:17.557 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:17.557 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:17.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:17.593 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:17.593+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:17.596 INFO:tasks.rados.rados.0.smithi181.stdout:111: expect (ObjNum 8 snap 0 seq_num 8) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:109: done (6 left) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:111: done (5 left) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:113: done (4 left) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:115: done (3 left) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:116: done (2 left) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:117: done (1 left) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:118: done (0 left) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:119: read oid 43 snap -1 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:119: expect (ObjNum 58 snap 4 seq_num 58) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:120: read oid 2 snap -1 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:120: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 19 current snap is 5 2024-04-26T01:24:17.608 INFO:tasks.rados.rados.0.smithi181.stdout:121: seq_num 62 ranges {311296=671744,983040=376832} 2024-04-26T01:24:17.623 INFO:tasks.rados.rados.0.smithi181.stdout:121: writing smithi18128752-19 from 311296 to 983040 tid 1 2024-04-26T01:24:17.630 INFO:tasks.rados.rados.0.smithi181.stdout:121: writing smithi18128752-19 from 983040 to 1359872 tid 2 2024-04-26T01:24:17.630 INFO:tasks.rados.rados.0.smithi181.stdout:122: snap_create 2024-04-26T01:24:17.694 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:17.696+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:17.739 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:17.739 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:17.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:146 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.577085943, 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.841318758, 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:17.740 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 146 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:17.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165148+0000", 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6000000000000001e-06 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165979+0000", 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00083094699999999996 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166013+0000", 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3386000000000002e-05 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166028+0000", 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5480000000000001e-05 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189275+0000", 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.023246692999999999 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189585+0000", 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031012999999999999 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.190986+0000", 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014009820000000001 2024-04-26T01:24:17.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295619+0000", 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104633299 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005890+0000", 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71027072300000005 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006464+0000", 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057451799999999999 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:147 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.5708465, 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93684057600000004, 2024-04-26T01:24:17.746 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 147 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:17.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171387+0000", 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4099999999999998e-06 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171984+0000", 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059681399999999998 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172016+0000", 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1779000000000002e-05 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172031+0000", 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5026e-05 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192109+0000", 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.020077514000000001 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192330+0000", 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00022101 2024-04-26T01:24:17.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.193866+0000", 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001536239 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295801+0000", 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101934767 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.107290+0000", 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81148932200000001 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108226+0000", 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093569499999999995 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:148 3.es0 3:734f3d85:::smithi18128752-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.570172648, 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.137414704, 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.753 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 148 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:17.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172060+0000", 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.513e-06 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172069+0000", 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0930000000000003e-06 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172088+0000", 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9303000000000001e-05 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172103+0000", 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5586e-05 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496369+0000", 2024-04-26T01:24:17.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32426532299999999 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496614+0000", 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024487400000000001 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.497656+0000", 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010425289999999999 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.599811+0000", 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10215491 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.308958+0000", 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70914692000000001 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.309474+0000", 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051565300000000003 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:149 3.es0 3:734f3d85:::smithi18128752-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.570092341000001, 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.54710463, 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.759 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 149 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172141+0000", 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.39e-06 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172148+0000", 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0600000000000002e-06 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172161+0000", 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3803e-05 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172171+0000", 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4069999999999999e-06 2024-04-26T01:24:17.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498185+0000", 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.326014476 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498406+0000", 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000220461 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311032+0000", 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81262658200000004 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311366+0000", 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00033330099999999999 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.719244+0000", 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40787814999999999 2024-04-26T01:24:17.764 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.282885998999999, 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0565503999999999, 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.765 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9999999999999998e-07 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459365+0000", 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8600000000000001e-05 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459378+0000", 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2989e-05 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459388+0000", 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0710000000000001e-05 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515199+0000", 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.055811056 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515896+0000", 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069664499999999997 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.281304484, 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54462607299999999, 2024-04-26T01:24:17.769 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460929+0000", 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.72e-06 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460937+0000", 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.763e-06 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460965+0000", 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8693000000000001e-05 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460979+0000", 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4072e-05 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005512+0000", 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54453236000000005 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005553+0000", 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1464999999999997e-05 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.773 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.280689011, 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54510908400000002, 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461544+0000", 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461554+0000", 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3700000000000001e-06 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461582+0000", 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8062e-05 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461596+0000", 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.432e-05 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006604+0000", 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54500755300000003 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006652+0000", 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8041999999999998e-05 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.266810811999999, 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0397411919999999, 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:17.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475422+0000", 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475446+0000", 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4428999999999999e-05 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475478+0000", 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1516000000000001e-05 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.575602+0000", 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100124637 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515084+0000", 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93948142300000004 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515162+0000", 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8213999999999993e-05 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.781 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.242809588, 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60899775199999995, 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499424+0000", 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7630000000000001e-06 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499436+0000", 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.201e-05 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499465+0000", 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9142999999999999e-05 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499479+0000", 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4419e-05 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108343+0000", 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60886387600000003 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108420+0000", 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6540999999999998e-05 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=20, to_read={3:7f6c3760:::smithi18128752-23:head=0,4096,0}, subchunks={3:7f6c3760:::smithi18128752-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.141044133999999, 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91354819600000003, 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.786 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.788 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601264+0000", 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6395000000000003e-05 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601293+0000", 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8829000000000001e-05 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514183+0000", 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91289044600000002 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514736+0000", 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055252600000000002 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.140031223999999, 2024-04-26T01:24:17.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.113522984, 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602202+0000", 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602223+0000", 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1036e-05 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602251+0000", 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7773e-05 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702368+0000", 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100117531 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715639+0000", 2024-04-26T01:24:17.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0132701660000001 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715723+0000", 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4738000000000001e-05 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:17.794 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.039766923, 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81161888599999998, 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702467+0000", 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0370000000000001e-06 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702492+0000", 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5096000000000002e-05 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702519+0000", 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7152e-05 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702533+0000", 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4464000000000001e-05 2024-04-26T01:24:17.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514000+0000", 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81146615499999997 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514084+0000", 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3981999999999997e-05 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.798 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.964231361, 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83750464400000002, 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778002+0000", 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.64e-06 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778026+0000", 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778058+0000", 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2398999999999999e-05 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778076+0000", 2024-04-26T01:24:17.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7408999999999999e-05 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615429+0000", 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83735304799999999 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615505+0000", 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6057999999999996e-05 2024-04-26T01:24:17.802 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=21, to_read={3:7e2e4071:::smithi18128752-29:head=0,4096,0}, subchunks={3:7e2e4071:::smithi18128752-29:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.837042123, 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018501300000003, 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.803 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:17.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:17.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905272+0000", 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2304000000000003e-05 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905302+0000", 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9850000000000001e-05 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615005+0000", 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70970295900000002 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615374+0000", 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036989999999999999 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.836121523999999, 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60991441499999999, 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:17.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906112+0000", 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.176e-06 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906133+0000", 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1135999999999999e-05 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906161+0000", 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7783000000000001e-05 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906175+0000", 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3419e-05 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515955+0000", 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60978075899999995 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.516024+0000", 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9141999999999996e-05 2024-04-26T01:24:17.811 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.736077249999999, 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71033649600000004, 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.812 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006156+0000", 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006182+0000", 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6262999999999999e-05 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006213+0000", 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0298999999999999e-05 2024-04-26T01:24:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006233+0000", 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0169e-05 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716419+0000", 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018600300000001 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716491+0000", 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2132000000000007e-05 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.735842397000001, 2024-04-26T01:24:17.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71023971900000005, 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:17.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006391+0000", 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006420+0000", 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8932999999999999e-05 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006451+0000", 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1086e-05 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006469+0000", 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8016000000000002e-05 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716554+0000", 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71008492899999998 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716629+0000", 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4857999999999994e-05 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.820 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:254 3.es0 3:7f6c3760:::smithi18128752-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.5453579680000002, 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71203804900000001, 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 254 2024-04-26T01:24:17.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9299999999999997e-07 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196880+0000", 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2410000000000003e-06 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.823 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196890+0000", 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0142999999999999e-05 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.296984+0000", 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100093949 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303480+0000", 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.006496646 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303606+0000", 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012540699999999999 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706770+0000", 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40316384799999999 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:17.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706914+0000", 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000144666 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.908912+0000", 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20199735599999999 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:257 3.es0 3:7915a690:::smithi18128752-22:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:17.826 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.4437027520000001, 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71064416799999996, 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 257 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298530+0000", 2024-04-26T01:24:17.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7999999999999993e-07 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298537+0000", 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8700000000000003e-06 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298551+0000", 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4202999999999999e-05 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298559+0000", 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0730000000000005e-06 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709349+0000", 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41079035699999999 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:17.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709598+0000", 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024883099999999998 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.710488+0000", 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00088957200000000004 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.811226+0000", 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100738614 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.008990+0000", 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19776387300000001 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009173+0000", 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000182795 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:258 3.es0 3:7915a690:::smithi18128752-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.4436614060000004, 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.99510092800000005, 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 258 2024-04-26T01:24:17.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298571+0000", 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298585+0000", 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4046000000000001e-05 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298599+0000", 2024-04-26T01:24:17.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4192999999999999e-05 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298608+0000", 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6030000000000005e-06 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712320+0000", 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41371240100000001 2024-04-26T01:24:17.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712472+0000", 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000151586 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009806+0000", 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.297334565 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:17.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.010023+0000", 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000216698 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.293671+0000", 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28364788600000002 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.838 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:17.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:17.841 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:17.840+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:17.867 INFO:tasks.rados.rados.0.smithi181.stdout:119: expect (ObjNum 58 snap 4 seq_num 58) 2024-04-26T01:24:17.941 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:17.943+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:17.970 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:17.970 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=68, reqid=client.4589.0:313, at_version=46'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.012324284, 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.012362524, 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963902+0000", 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964264+0000", 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036215900000000003 2024-04-26T01:24:17.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964306+0000", 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1801999999999997e-05 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964334+0000", 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8133e-05 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968462+0000", 2024-04-26T01:24:17.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041272710000000001 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968486+0000", 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4219999999999999e-05 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=69, reqid=client.4589.0:314, at_version=46'19, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0081613439999999992, 2024-04-26T01:24:17.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0083038910000000007, 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968064+0000", 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9829999999999999e-06 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968174+0000", 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000109028 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968220+0000", 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6106000000000002e-05 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968244+0000", 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4239e-05 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970652+0000", 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024076810000000001 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970665+0000", 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3652000000000001e-05 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=70, reqid=client.4589.0:315, at_version=46'20, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.005283187, 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0055210290000000002, 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:17.978 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970942+0000", 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8339999999999999e-06 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971003+0000", 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0401e-05 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971040+0000", 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7469999999999999e-05 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:17.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971064+0000", 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3609e-05 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 3 2024-04-26T01:24:17.981 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:17.985 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:18.014 INFO:tasks.rados.rados.0.smithi181.stdout:120: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-26T01:24:18.042 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:18.044+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:18.103 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:18.103 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:18.103 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:18.103 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:18.103 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:18.117 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:18.143 INFO:tasks.rados.rados.0.smithi181.stdout:119: done (3 left) 2024-04-26T01:24:18.143 INFO:tasks.rados.rados.0.smithi181.stdout:120: done (2 left) 2024-04-26T01:24:18.143 INFO:tasks.rados.rados.0.smithi181.stdout:122: done (1 left) 2024-04-26T01:24:18.143 INFO:tasks.rados.rados.0.smithi181.stdout:123: read oid 4 snap 5 2024-04-26T01:24:18.143 INFO:tasks.rados.rados.0.smithi181.stdout:123: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:24:18.143 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:18.141+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:18.144 INFO:tasks.rados.rados.0.smithi181.stdout:124: read oid 35 snap 2 2024-04-26T01:24:18.144 INFO:tasks.rados.rados.0.smithi181.stdout:124: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-26T01:24:18.144 INFO:tasks.rados.rados.0.smithi181.stdout:125: delete oid 34 current snap is 6 2024-04-26T01:24:18.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:24:18.220 INFO:tasks.rados.rados.0.smithi181.stdout:121: finishing write tid 1 to smithi18128752-19 2024-04-26T01:24:18.221 INFO:tasks.rados.rados.0.smithi181.stdout:121: finishing write tid 2 to smithi18128752-19 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 35/34 ECSubWrite(tid=5, reqid=client.4589.0:116, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.683710522, 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20106410399999999, 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.240 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.241 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559045+0000", 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8929999999999999e-06 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559097+0000", 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2672e-05 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559128+0000", 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1019000000000002e-05 2024-04-26T01:24:18.242 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659233+0000", 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100104779 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659393+0000", 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016010199999999999 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659420+0000", 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6333e-05 2024-04-26T01:24:18.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760047+0000", 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100627094 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760107+0000", 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0211999999999999e-05 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.244 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s3 35/30 ECSubWrite(tid=166, reqid=client.4589.0:130, at_version=35'5, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.659588+0000", 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.583165462, 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102237551, 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659588+0000", 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.245 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659588+0000", 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659588+0000", 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1300000000000001e-07 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659773+0000", 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018466899999999999 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.246 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659788+0000", 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4816e-05 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659801+0000", 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3253e-05 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659957+0000", 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000156613 2024-04-26T01:24:18.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659981+0000", 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3448999999999999e-05 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.761779+0000", 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101798483 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.761825+0000", 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5955000000000003e-05 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s3 35/30 ECSubWrite(tid=167, reqid=client.4589.0:131, at_version=35'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.659832+0000", 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.582921032, 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10299027099999999, 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.249 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659832+0000", 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659832+0000", 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.250 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659833+0000", 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659877+0000", 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3801999999999998e-05 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659897+0000", 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0625999999999999e-05 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.251 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659916+0000", 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8510000000000001e-05 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.761482+0000", 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101566559 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.761497+0000", 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4676e-05 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.762751+0000", 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012543330000000001 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.762823+0000", 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1118000000000006e-05 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/34 ECSubWrite(tid=9, reqid=client.4589.0:71, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.660109+0000", 2024-04-26T01:24:18.253 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.582644629000001, 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102039176, 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.660109+0000", 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.660109+0000", 2024-04-26T01:24:18.254 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.660110+0000", 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0270000000000001e-06 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.660143+0000", 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3479000000000003e-05 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.255 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.660165+0000", 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2052000000000002e-05 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.660184+0000", 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8649999999999999e-05 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760464+0000", 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10028015 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760482+0000", 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7825999999999998e-05 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.762080+0000", 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0015986240000000001 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.762148+0000", 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7367999999999996e-05 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.257 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 35/30 ECSubWrite(tid=34, reqid=client.4589.0:134, at_version=35'16, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.835499+0000", 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.407253797999999, 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10232667199999999, 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.258 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.835499+0000", 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.835499+0000", 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.835500+0000", 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6300000000000004e-07 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.259 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.835519+0000", 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8966000000000001e-05 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.835530+0000", 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0686000000000001e-05 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.835543+0000", 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2687000000000001e-05 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.936441+0000", 2024-04-26T01:24:18.261 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10089859399999999 2024-04-26T01:24:18.261 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.936459+0000", 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7725999999999999e-05 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.262 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.937733+0000", 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012745689999999999 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.937826+0000", 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2681000000000003e-05 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.es5 35/34 ECSubWrite(tid=10, reqid=client.4589.0:65, at_version=35'4, trim_to=0'0, roll_forward_to=35'3))", 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.869364+0000", 2024-04-26T01:24:18.263 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.373389103999999, 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101790477, 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.869364+0000", 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.869364+0000", 2024-04-26T01:24:18.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.869366+0000", 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0200000000000001e-06 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.869414+0000", 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7675e-05 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.869466+0000", 2024-04-26T01:24:18.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1965999999999997e-05 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.969606+0000", 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100140391 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.969816+0000", 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00020950499999999999 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.969840+0000", 2024-04-26T01:24:18.266 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4451999999999999e-05 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.971058+0000", 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012174449999999999 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.971155+0000", 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7022999999999996e-05 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/34 ECSubWrite(tid=15, reqid=client.4589.0:76, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.159076+0000", 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.083676998, 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102575243, 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.159076+0000", 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.159076+0000", 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.159078+0000", 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.767e-06 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.159301+0000", 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00022333100000000001 2024-04-26T01:24:18.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.159332+0000", 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0255000000000001e-05 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.159347+0000", 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5506999999999999e-05 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.259644+0000", 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100297147 2024-04-26T01:24:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.259659+0000", 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4412e-05 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.261565+0000", 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001906519 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.261652+0000", 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6304999999999994e-05 2024-04-26T01:24:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs4 35/34 ECSubWrite(tid=21, reqid=client.4589.0:151, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.274014+0000", 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.968739673, 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102846416, 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.274014+0000", 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.274014+0000", 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.274016+0000", 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.95e-06 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.274295+0000", 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000279726 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.274348+0000", 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3050999999999997e-05 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.274373+0000", 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4723e-05 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.374681+0000", 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100307471 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.374713+0000", 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2449000000000001e-05 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.376806+0000", 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002092614 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.376860+0000", 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4432000000000003e-05 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 36/34 ECSubWrite(tid=14, reqid=client.4589.0:171, at_version=36'8, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.338894+0000", 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.903859513, 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101770451, 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.276 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.338894+0000", 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.338894+0000", 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.338896+0000", 2024-04-26T01:24:18.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.338930+0000", 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4978999999999998e-05 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.338960+0000", 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9736000000000001e-05 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.338983+0000", 2024-04-26T01:24:18.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2785999999999998e-05 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.439322+0000", 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10033877500000001 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.439346+0000", 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4247e-05 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.440601+0000", 2024-04-26T01:24:18.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012553760000000001 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.440664+0000", 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2811999999999994e-05 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/30 ECSubWrite(tid=172, reqid=client.4589.0:167, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.407368+0000", 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.835385283000001, 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104738489, 2024-04-26T01:24:18.280 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407368+0000", 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407368+0000", 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407369+0000", 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.237e-06 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407683+0000", 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031392099999999999 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407712+0000", 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8493e-05 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.507844+0000", 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100132422 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.508011+0000", 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000167355 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.508041+0000", 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9280000000000001e-05 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.512030+0000", 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0039888359999999999 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.512106+0000", 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6945000000000003e-05 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 37/30 ECSubWrite(tid=25, reqid=client.4589.0:188, at_version=37'14, trim_to=0'0, roll_forward_to=35'12))", 2024-04-26T01:24:18.284 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.182577+0000", 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.060176186, 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102732974, 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182577+0000", 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182577+0000", 2024-04-26T01:24:18.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182578+0000", 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9299999999999998e-07 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182816+0000", 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023802799999999999 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182852+0000", 2024-04-26T01:24:18.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6055e-05 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.282953+0000", 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10010087500000001 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283073+0000", 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000120294 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283088+0000", 2024-04-26T01:24:18.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4780000000000001e-05 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.285246+0000", 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021585810000000001 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.285310+0000", 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3768000000000003e-05 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.bs4 37/34 ECSubWrite(tid=55, reqid=client.4589.0:207, at_version=37'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.283250+0000", 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.9595033550000007, 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10229049599999999, 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283250+0000", 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283250+0000", 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283251+0000", 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3200000000000001e-06 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283337+0000", 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5814000000000002e-05 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283361+0000", 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4043e-05 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.283382+0000", 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0775999999999999e-05 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.383594+0000", 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100212043 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.383609+0000", 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5369000000000001e-05 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.385469+0000", 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018592789999999999 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.385540+0000", 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1852000000000003e-05 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.as5 40/30 ECSubWrite(tid=11, reqid=client.4589.0:264, at_version=0'0, trim_to=0'0, roll_forward_to=40'9))", 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.608467+0000", 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.6342864529999996, 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10178527900000001, 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.608467+0000", 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.608467+0000", 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.608469+0000", 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8929999999999999e-06 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.608498+0000", 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9203000000000001e-05 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.608540+0000", 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1955000000000001e-05 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.708656+0000", 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10011608600000001 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.708856+0000", 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00020047500000000001 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.708881+0000", 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4182999999999999e-05 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.710142+0000", 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001261554 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.710252+0000", 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010993 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.296 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.es5 40/34 ECSubWrite(tid=40, reqid=client.4589.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=40'23))", 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.2332793070000001, 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30174757699999999, 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:18.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009476+0000", 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9800000000000001e-06 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.298 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009502+0000", 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6495999999999998e-05 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009541+0000", 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8748999999999998e-05 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009564+0000", 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2773e-05 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109788+0000", 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100224176 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109803+0000", 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4487e-05 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311086+0000", 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201283252 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311222+0000", 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013566399999999999 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 41/34 ECSubWrite(tid=27, reqid=osd.0.0:27, at_version=41'17, trim_to=0'0, roll_forward_to=39'15))", 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.1433057780000002, 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21182642600000001, 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7300000000000001e-07 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099485+0000", 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7493000000000002e-05 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099509+0000", 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.387e-05 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099527+0000", 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7598999999999999e-05 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110237+0000", 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.010710366000000001 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110253+0000", 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5319999999999999e-05 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311234+0000", 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20098134600000001 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311274+0000", 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9959000000000001e-05 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=32, reqid=client.4589.0:266, at_version=41'17, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.0676072459999997, 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13212590199999999, 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.305 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175147+0000", 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.13e-07 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175731+0000", 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000583973 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175761+0000", 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0536e-05 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175777+0000", 2024-04-26T01:24:18.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6076e-05 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302772+0000", 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12699442499999999 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302793+0000", 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1382999999999998e-05 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307218+0000", 2024-04-26T01:24:18.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0044247139999999997 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307272+0000", 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4181999999999997e-05 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=33, reqid=osd.5.0:33, at_version=41'19, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.0646348640000003, 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.129420758, 2024-04-26T01:24:18.309 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178120+0000", 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178160+0000", 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9356000000000001e-05 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178190+0000", 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0179e-05 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178206+0000", 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5809000000000001e-05 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.304772+0000", 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12656614899999999 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.304787+0000", 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4893e-05 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307500+0000", 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027131490000000002 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307539+0000", 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9486000000000001e-05 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=34, reqid=client.4589.0:267, at_version=41'20, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.181203+0000", 2024-04-26T01:24:18.313 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.0615505230000002, 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.127937575, 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181203+0000", 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181203+0000", 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181205+0000", 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181296+0000", 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1978000000000006e-05 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181326+0000", 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9998999999999999e-05 2024-04-26T01:24:18.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.281469+0000", 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10014216500000001 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.306125+0000", 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.024655928000000001 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.306138+0000", 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3003000000000001e-05 2024-04-26T01:24:18.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309045+0000", 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029073609999999998 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309140+0000", 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5397999999999997e-05 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs4 45/34 ECSubWrite(tid=42, reqid=client.4589.0:297, at_version=45'21, trim_to=0'0, roll_forward_to=45'19))", 2024-04-26T01:24:18.317 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:16.751305+0000", 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.491447899, 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10363475599999999, 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751305+0000", 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751305+0000", 2024-04-26T01:24:18.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751307+0000", 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.593e-06 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751714+0000", 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00040656500000000001 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751741+0000", 2024-04-26T01:24:18.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7239000000000001e-05 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751761+0000", 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0486e-05 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.852002+0000", 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100241011 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.852034+0000", 2024-04-26T01:24:18.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1655000000000001e-05 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.854873+0000", 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028389719999999999 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.854940+0000", 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7235000000000003e-05 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.8s6 46/44 ECSubRead(tid=45, to_read={3:18914e2a:::smithi18128752-43:head=0,425984,0}, subchunks={3:18914e2a:::smithi18128752-43:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.751971+0000", 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.49078223599999998, 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10189801599999999, 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.751971+0000", 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.751971+0000", 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.751972+0000", 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1999999999999999e-07 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.751983+0000", 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1164e-05 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.752008+0000", 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5562e-05 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.852118+0000", 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10010975900000001 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.853452+0000", 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001333849 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.853869+0000", 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000416962 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.325 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:18.326 INFO:tasks.rados.rados.0.smithi181.stdout:121: finishing write tid 3 to smithi18128752-19 2024-04-26T01:24:18.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:18.327 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:18.328+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:18.355 INFO:tasks.rados.rados.0.smithi181.stdout:125: done (3 left) 2024-04-26T01:24:18.355 INFO:tasks.rados.rados.0.smithi181.stdout:126: rollback oid 21 current snap is 6 2024-04-26T01:24:18.355 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 21 to 2 2024-04-26T01:24:18.355 INFO:tasks.rados.rados.0.smithi181.stdout:127: read oid 17 snap -1 2024-04-26T01:24:18.356 INFO:tasks.rados.rados.0.smithi181.stdout:127: expect (ObjNum 55 snap 3 seq_num 55) 2024-04-26T01:24:18.356 INFO:tasks.rados.rados.0.smithi181.stdout:128: setattr oid 20 current snap is 6 2024-04-26T01:24:18.361 INFO:tasks.rados.rados.0.smithi181.stdout:129: delete oid 10 current snap is 6 2024-04-26T01:24:18.427 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:18.429+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:18.498 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:18.498 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:18.498 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:18.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:18.528 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:18.531+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:18.553 INFO:tasks.rados.rados.0.smithi181.stdout:123: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:24:18.557 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 8 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T01:24:18.571 INFO:tasks.rados.rados.0.smithi181.stdout:121: finishing write tid 4 to smithi18128752-19 2024-04-26T01:24:18.571 INFO:tasks.rados.rados.0.smithi181.stdout:121: oid 19 updating version 0 to 19 2024-04-26T01:24:18.571 INFO:tasks.rados.rados.0.smithi181.stdout:121: oid 19 version 19 is already newer than 18 2024-04-26T01:24:18.572 INFO:tasks.rados.rados.0.smithi181.stdout:121: oid 19 updating version 19 to 20 2024-04-26T01:24:18.572 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 20 (ObjNum 62 snap 5 seq_num 62) dirty exists 2024-04-26T01:24:18.572 INFO:tasks.rados.rados.0.smithi181.stdout:121: left oid 19 (ObjNum 62 snap 5 seq_num 62) 2024-04-26T01:24:18.629 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:18.631+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:18.635 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:18.635 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:18.635 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:18.635 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:18.635 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:18.650 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:18.730 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:18.733+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:18.773 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:18.773 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:18.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:18.773 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:18.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.773 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/30 ECSubWrite(tid=40, reqid=client.4589.0:137, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:18.773 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.783636+0000", 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.993919731, 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100477629, 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.783636+0000", 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.783636+0000", 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.783637+0000", 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1200000000000001e-06 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.783670+0000", 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2728999999999998e-05 2024-04-26T01:24:18.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.783690+0000", 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0908999999999999e-05 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.783706+0000", 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.504e-05 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.883046+0000", 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.099340125000000001 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.883058+0000", 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2116999999999999e-05 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.884032+0000", 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00097389099999999995 2024-04-26T01:24:18.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.884113+0000", 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1698000000000003e-05 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/30 ECSubWrite(tid=42, reqid=client.4589.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.087029+0000", 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.690526307000001, 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101532873, 2024-04-26T01:24:18.778 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087029+0000", 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087029+0000", 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087031+0000", 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9130000000000001e-06 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087054+0000", 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2688999999999999e-05 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087083+0000", 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8856999999999998e-05 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087098+0000", 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5628999999999999e-05 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.187328+0000", 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100230262 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.187344+0000", 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5458999999999999e-05 2024-04-26T01:24:18.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188505+0000", 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011609109999999999 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188562+0000", 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7153e-05 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/30 ECSubWrite(tid=44, reqid=client.4589.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.188045+0000", 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.589510827, 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10172057599999999, 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188045+0000", 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188045+0000", 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188047+0000", 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1600000000000001e-06 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188078+0000", 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0852999999999998e-05 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188110+0000", 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2026000000000001e-05 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188152+0000", 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1919000000000003e-05 2024-04-26T01:24:18.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.288563+0000", 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10041122299999999 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.288593+0000", 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0369e-05 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.289697+0000", 2024-04-26T01:24:18.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001103971 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.289765+0000", 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8054999999999998e-05 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/30 ECSubWrite(tid=47, reqid=client.4589.0:140, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.484636502000001, 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20188384300000001, 2024-04-26T01:24:18.787 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.788 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.788 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.788 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292921+0000", 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8870000000000001e-06 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292964+0000", 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3152000000000003e-05 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.293004+0000", 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0065999999999999e-05 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.393103+0000", 2024-04-26T01:24:18.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100099122 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493412+0000", 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10030921299999999 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493428+0000", 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494705+0000", 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012777590000000001 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494803+0000", 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7477000000000002e-05 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/30 ECSubWrite(tid=39, reqid=client.4589.0:154, at_version=35'7, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.662714+0000", 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.114841375999999, 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10209053999999999, 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662714+0000", 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662714+0000", 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662716+0000", 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.187e-06 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662795+0000", 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9223999999999999e-05 2024-04-26T01:24:18.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662827+0000", 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1678999999999999e-05 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662843+0000", 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6223e-05 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.763113+0000", 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100269335 2024-04-26T01:24:18.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.763132+0000", 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.893e-05 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764708+0000", 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001576647 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764805+0000", 2024-04-26T01:24:18.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6315000000000002e-05 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/34 ECSubWrite(tid=18, reqid=client.4589.0:77, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.764464+0000", 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.013091631, 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101881215, 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764464+0000", 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764464+0000", 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764466+0000", 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8139999999999999e-06 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764490+0000", 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4304999999999999e-05 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764517+0000", 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6869999999999999e-05 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764541+0000", 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4629000000000001e-05 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764694+0000", 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015242599999999999 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764718+0000", 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4026e-05 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.866267+0000", 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101548914 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.866345+0000", 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8231000000000004e-05 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.cs3 35/34 ECSubRead(tid=19, to_read={3:3a8fbb99:::smithi18128752-26:head=0,4096,0}, subchunks={3:3a8fbb99:::smithi18128752-26:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.765324+0000", 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.012231896999999, 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100935946, 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.765324+0000", 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.765324+0000", 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.765325+0000", 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3969999999999999e-06 2024-04-26T01:24:18.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.765341+0000", 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6158999999999999e-05 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.765370+0000", 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9352999999999998e-05 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.765388+0000", 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7356000000000001e-05 2024-04-26T01:24:18.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.865670+0000", 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100282441 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.866259+0000", 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058923999999999999 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 36/34 ECSubWrite(tid=26, reqid=client.4589.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:18.804 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.097261026, 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200889605, 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680296+0000", 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8640000000000001e-06 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680320+0000", 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4202e-05 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680350+0000", 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9998999999999999e-05 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780494+0000", 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100143549 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780762+0000", 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026773600000000003 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780776+0000", 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:24:18.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881147+0000", 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100370127 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881184+0000", 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7422000000000002e-05 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.cs3 36/34 ECSubRead(tid=27, to_read={3:3a0c1dcc:::smithi18128752-50:head=0,4096,0}, subchunks={3:3a0c1dcc:::smithi18128752-50:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:18.808 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.780549+0000", 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.997006401, 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10058948199999999, 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780549+0000", 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780549+0000", 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780551+0000", 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7829999999999999e-06 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780569+0000", 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8559999999999998e-05 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780600+0000", 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0589000000000001e-05 2024-04-26T01:24:18.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780637+0000", 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7029000000000001e-05 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.880757+0000", 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100120028 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881139+0000", 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000381493 2024-04-26T01:24:18.811 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.ds3 37/30 ECSubRead(tid=56, to_read={3:b136579b:::smithi18128752-36:head=0,147456,0}, subchunks={3:b136579b:::smithi18128752-36:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.260592+0000", 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.516963027999999, 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10135630700000001, 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.260592+0000", 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.260592+0000", 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.260593+0000", 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9699999999999996e-07 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.260607+0000", 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.431e-05 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.260645+0000", 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7635999999999997e-05 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.260672+0000", 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7379e-05 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.814 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361018+0000", 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100345406 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361949+0000", 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093107900000000002 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.815 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.ds3 37/30 ECSubRead(tid=57, to_read={3:b136579b:::smithi18128752-36:head=0,147456,0}, subchunks={3:b136579b:::smithi18128752-36:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.361267+0000", 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.416288003, 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101099956, 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361267+0000", 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.816 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361267+0000", 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361270+0000", 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0999999999999998e-06 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361287+0000", 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7258999999999999e-05 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.817 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361321+0000", 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4289000000000003e-05 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.361342+0000", 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1223e-05 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.462167+0000", 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100824837 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.818 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.462367+0000", 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00020024800000000001 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 37/30 ECSubWrite(tid=45, reqid=client.4589.0:196, at_version=37'9, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.378646+0000", 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.398909567, 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10144679400000001, 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.819 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378646+0000", 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378646+0000", 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378646+0000", 2024-04-26T01:24:18.820 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5299999999999999e-07 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378709+0000", 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2688999999999996e-05 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378732+0000", 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3456e-05 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378753+0000", 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0879000000000001e-05 2024-04-26T01:24:18.821 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378868+0000", 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000114864 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378883+0000", 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.507e-05 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480043+0000", 2024-04-26T01:24:18.822 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101160008 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480093+0000", 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9375000000000001e-05 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 37/30 ECSubWrite(tid=46, reqid=client.4589.0:197, at_version=37'10, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.378788+0000", 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.398767288, 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10133714100000001, 2024-04-26T01:24:18.823 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378788+0000", 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378788+0000", 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.824 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378789+0000", 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6400000000000002e-07 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378813+0000", 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.393e-05 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378826+0000", 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2816000000000001e-05 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.825 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378836+0000", 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0243e-05 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.379634+0000", 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00079825600000000001 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.379649+0000", 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5089000000000001e-05 2024-04-26T01:24:18.826 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480099+0000", 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100449494 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480125+0000", 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6648999999999998e-05 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.827 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 37/30 ECSubWrite(tid=47, reqid=client.4589.0:198, at_version=37'11, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.398696479, 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10198423299999999, 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:18.828 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8299999999999997e-07 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378883+0000", 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3235999999999999e-05 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.829 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378894+0000", 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1579999999999999e-05 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378903+0000", 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2059999999999996e-06 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.479752+0000", 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100848803 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.830 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.479771+0000", 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8383e-05 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480783+0000", 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010124540000000001 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480843+0000", 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0087999999999998e-05 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.831 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/30 ECSubWrite(tid=90, reqid=osd.4.0:90, at_version=41'10, trim_to=0'0, roll_forward_to=37'8))", 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.5779023250000002, 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103318356, 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:18.832 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199654+0000", 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199704+0000", 2024-04-26T01:24:18.833 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9656e-05 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199735+0000", 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1396000000000002e-05 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199762+0000", 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7092000000000001e-05 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.299984+0000", 2024-04-26T01:24:18.834 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100221583 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.300000+0000", 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5906999999999999e-05 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302870+0000", 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028700879999999998 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.835 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302971+0000", 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010186799999999999 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/34 ECSubWrite(tid=33, reqid=osd.11.0:33, at_version=41'13, trim_to=0'0, roll_forward_to=37'11))", 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.207265+0000", 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.5702901730000001, 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101853824, 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.836 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207265+0000", 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207265+0000", 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207266+0000", 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4299999999999999e-07 2024-04-26T01:24:18.837 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207334+0000", 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8591999999999997e-05 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207357+0000", 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2943e-05 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207373+0000", 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5486e-05 2024-04-26T01:24:18.838 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307636+0000", 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100263175 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307654+0000", 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8627e-05 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309043+0000", 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001388303 2024-04-26T01:24:18.839 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309119+0000", 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6354999999999997e-05 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.375513196, 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201895835, 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.840 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.841 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402045+0000", 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2800000000000002e-06 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402074+0000", 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9366e-05 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402148+0000", 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4255e-05 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.842 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502254+0000", 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100106322 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502424+0000", 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000169259 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502449+0000", 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4909000000000002e-05 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.843 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603856+0000", 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101407649 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603938+0000", 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1794999999999999e-05 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/30 ECSubWrite(tid=92, reqid=osd.4.0:90, at_version=0'0, trim_to=0'0, roll_forward_to=41'10))", 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.502715+0000", 2024-04-26T01:24:18.844 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.2748399739999998, 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10209800300000001, 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502715+0000", 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502715+0000", 2024-04-26T01:24:18.845 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502717+0000", 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.606e-06 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502742+0000", 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5409e-05 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.846 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502770+0000", 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7540000000000001e-05 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502793+0000", 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2739e-05 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603415+0000", 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100622239 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.847 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603444+0000", 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9224999999999999e-05 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.604744+0000", 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013000659999999999 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.604813+0000", 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9178999999999996e-05 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.848 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 45/34 ECSubWrite(tid=43, reqid=client.4589.0:297, at_version=0'0, trim_to=0'0, roll_forward_to=45'21))", 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.048297+0000", 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.7292588279999999, 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101580799, 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048297+0000", 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.849 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048297+0000", 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048299+0000", 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.04e-06 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048323+0000", 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4413000000000002e-05 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.850 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048355+0000", 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1619000000000003e-05 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048372+0000", 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.732e-05 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.148664+0000", 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100291619 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.851 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.148680+0000", 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6475999999999999e-05 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.149792+0000", 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001111807 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.149877+0000", 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5505000000000002e-05 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.852 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 46/44 ECSubWrite(tid=114, reqid=client.4589.0:304, at_version=0'0, trim_to=0'0, roll_forward_to=45'14))", 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.258768+0000", 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.518787356, 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101055041, 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.258768+0000", 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.853 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.258768+0000", 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.258769+0000", 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7e-07 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.258799+0000", 2024-04-26T01:24:18.854 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0357000000000001e-05 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.258831+0000", 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2011999999999998e-05 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.358934+0000", 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100103283 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.359100+0000", 2024-04-26T01:24:18.855 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016545899999999999 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.359114+0000", 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4324999999999999e-05 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.359796+0000", 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00068151699999999997 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.359823+0000", 2024-04-26T01:24:18.856 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7617999999999999e-05 2024-04-26T01:24:18.857 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.857 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.857 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.857 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.857 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.857 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:18.858 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:18.859 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:18.857+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:18.872 INFO:tasks.rados.rados.0.smithi181.stdout:121: done (6 left) 2024-04-26T01:24:18.872 INFO:tasks.rados.rados.0.smithi181.stdout:123: done (5 left) 2024-04-26T01:24:18.872 INFO:tasks.rados.rados.0.smithi181.stdout:128: done (4 left) 2024-04-26T01:24:18.872 INFO:tasks.rados.rados.0.smithi181.stdout:129: done (3 left) 2024-04-26T01:24:18.872 INFO:tasks.rados.rados.0.smithi181.stdout:130: snap_remove snap 5 2024-04-26T01:24:18.959 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:18.960+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:18.961 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:18.986 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:18.986 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:18.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.986 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 47/30 ECSubWriteReply(tid=75, last_complete=47'15, committed=1, applied=1))", 2024-04-26T01:24:18.986 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:18.880430+0000", 2024-04-26T01:24:18.986 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.11160748, 2024-04-26T01:24:18.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.11164661200000001, 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.880430+0000", 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.880430+0000", 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.880431+0000", 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0210000000000001e-06 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.880437+0000", 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9109999999999998e-06 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.880451+0000", 2024-04-26T01:24:18.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.385e-05 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.880461+0000", 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2709999999999998e-06 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDPGPushReply(3.0s0 47/44 [PushReplyOp(3:025aaed1:::smithi18128752-27:head)])", 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:18.991010+0000", 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.0010275760000000001, 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011577060000000001, 2024-04-26T01:24:18.989 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.991010+0000", 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.991010+0000", 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:18.990 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.991012+0000", 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9240000000000001e-06 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.991023+0000", 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0563e-05 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.991054+0000", 2024-04-26T01:24:18.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0886000000000002e-05 2024-04-26T01:24:18.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.992 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:18.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.992 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:18.992 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-04-26T01:24:18.992 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 2 2024-04-26T01:24:18.992 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:19.002 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:19.060 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:19.063+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:19.126 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:19.126 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:19.126 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:19.127 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:19.127 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:19.127 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:19.139 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:19.140 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:24:19.144 INFO:tasks.rados.rados.0.smithi181.stdout:130: done (3 left) 2024-04-26T01:24:19.144 INFO:tasks.rados.rados.0.smithi181.stdout:131: setattr oid 50 current snap is 6 2024-04-26T01:24:19.145 INFO:tasks.rados.rados.0.smithi181.stdout:124: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-26T01:24:19.184 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:19.188+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:19.192 INFO:tasks.rados.rados.0.smithi181.stdout:126: finishing rollback tid 0 to smithi18128752-21 2024-04-26T01:24:19.192 INFO:tasks.rados.rados.0.smithi181.stdout:126: finishing rollback tid 1 to smithi18128752-21 2024-04-26T01:24:19.192 INFO:tasks.rados.rados.0.smithi181.stdout:126: finishing rollback tid 2 to smithi18128752-21 2024-04-26T01:24:19.192 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 21 v 30 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-04-26T01:24:19.192 INFO:tasks.rados.rados.0.smithi181.stdout:127: expect (ObjNum 55 snap 3 seq_num 55) 2024-04-26T01:24:19.245 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:19.246 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:19.250 INFO:tasks.rados.rados.0.smithi181.stdout:124: done (3 left) 2024-04-26T01:24:19.250 INFO:tasks.rados.rados.0.smithi181.stdout:126: done (2 left) 2024-04-26T01:24:19.250 INFO:tasks.rados.rados.0.smithi181.stdout:127: done (1 left) 2024-04-26T01:24:19.250 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 48 current snap is 6 2024-04-26T01:24:19.250 INFO:tasks.rados.rados.0.smithi181.stdout:132: seq_num 65 ranges {802816=720896,1523712=376832} 2024-04-26T01:24:19.262 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:19.261+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:19.270 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:19.270 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:19.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:19.270 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:19.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.270 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/30 ECSubWriteReply(tid=34, last_complete=35'16, committed=1, applied=1))", 2024-04-26T01:24:19.270 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.038419+0000", 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 13.233543103000001, 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24232500200000001, 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038419+0000", 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038419+0000", 2024-04-26T01:24:19.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038421+0000", 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9700000000000002e-06 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038432+0000", 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1516e-05 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038461+0000", 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8339000000000001e-05 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038475+0000", 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4317e-05 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.280717+0000", 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24224263400000001 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.280744+0000", 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6225999999999999e-05 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/30 ECSubWriteReply(tid=33, last_complete=35'15, committed=1, applied=1))", 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.038938+0000", 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 13.233024193, 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24038295300000001, 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:19.274 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038938+0000", 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038938+0000", 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038938+0000", 2024-04-26T01:24:19.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.96e-07 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038941+0000", 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9529999999999999e-06 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038953+0000", 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2096000000000001e-05 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038962+0000", 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6300000000000004e-06 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.279292+0000", 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24033020899999999 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.279321+0000", 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8668999999999999e-05 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/30 ECSubWriteReply(tid=34, last_complete=35'16, committed=1, applied=1))", 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.038971+0000", 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 13.232990410999999, 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.24211664799999999, 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038971+0000", 2024-04-26T01:24:19.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038971+0000", 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038972+0000", 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9999999999999998e-07 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038974+0000", 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5170000000000002e-06 2024-04-26T01:24:19.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038986+0000", 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1559000000000001e-05 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.038994+0000", 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3259999999999996e-06 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.281066+0000", 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.242071642 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.281088+0000", 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2204e-05 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:154 3.9s0 3:958f0a11:::smithi18128752-2:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:19.281 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.997798912, 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59153377200000001, 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 154 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274166+0000", 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.027e-06 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274180+0000", 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3723000000000001e-05 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274213+0000", 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3698999999999997e-05 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274229+0000", 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5928999999999999e-05 2024-04-26T01:24:19.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274415+0000", 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018537499999999999 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274584+0000", 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016943000000000001 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662522+0000", 2024-04-26T01:24:19.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38793827400000003 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.664271+0000", 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00174884 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.865553+0000", 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20128183599999999 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.865697+0000", 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014363900000000001 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:155 3.9s0 3:958f0a11:::smithi18128752-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.274263+0000", 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.997698594999999, 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38956048500000001, 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.287 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 155 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274263+0000", 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274263+0000", 2024-04-26T01:24:19.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274264+0000", 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1570000000000001e-06 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274268+0000", 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9330000000000003e-06 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274280+0000", 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2089999999999999e-05 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.374422+0000", 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100141931 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.374562+0000", 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013998299999999999 2024-04-26T01:24:19.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.374716+0000", 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000153659 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.663544+0000", 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28882788300000001 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.663677+0000", 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013306599999999999 2024-04-26T01:24:19.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.663824+0000", 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014678299999999999 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:196 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.19456214, 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61790984400000004, 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.292 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 196 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8169999999999999e-06 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077416+0000", 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3783e-05 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7209000000000002e-05 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4168999999999998e-05 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077630+0000", 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015196599999999999 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077649+0000", 2024-04-26T01:24:19.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9262999999999998e-05 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.181790+0000", 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10414122100000001 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182072+0000", 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00028194299999999999 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.183846+0000", 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001774184 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189006+0000", 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0051592809999999999 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695163+0000", 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50615769499999996 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.297 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695310+0000", 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014631299999999999 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:197 3.9s0 3:93dca175:::smithi18128752-10:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.194433719999999, 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61806578000000001, 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.298 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 197 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077529+0000", 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3239999999999999e-06 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077537+0000", 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0560000000000005e-06 2024-04-26T01:24:19.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077556+0000", 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8456000000000001e-05 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077573+0000", 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6566e-05 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077807+0000", 2024-04-26T01:24:19.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023464800000000001 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077822+0000", 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5206e-05 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184696+0000", 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10687403099999999 2024-04-26T01:24:19.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184837+0000", 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000141113 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.186190+0000", 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001352575 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.303 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189008+0000", 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028181450000000002 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695495+0000", 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50648651300000003 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695594+0000", 2024-04-26T01:24:19.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9147000000000005e-05 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:198 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.194367535, 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61826488800000001, 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.305 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 198 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:19.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077595+0000", 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.167e-06 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077601+0000", 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6500000000000001e-06 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077619+0000", 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7569000000000001e-05 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077643+0000", 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4652999999999999e-05 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077989+0000", 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034534799999999999 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078006+0000", 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7125999999999998e-05 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187076+0000", 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10906990799999999 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.309 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187274+0000", 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000198221 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.188283+0000", 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001008994 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189285+0000", 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001002221 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695760+0000", 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.506474914 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695859+0000", 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9117000000000007e-05 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:211 3.0s0 3:014d0a8d:::smithi18128752-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.078094+0000", 2024-04-26T01:24:19.311 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.193868058, 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41330118300000002, 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 211 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078094+0000", 2024-04-26T01:24:19.312 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078094+0000", 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078095+0000", 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4699999999999999e-06 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078103+0000", 2024-04-26T01:24:19.313 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1999999999999994e-06 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078122+0000", 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8445999999999999e-05 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.314 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078136+0000", 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4486e-05 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078272+0000", 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013548100000000001 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078289+0000", 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7309e-05 2024-04-26T01:24:19.315 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.083286+0000", 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0049972180000000003 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.083443+0000", 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000156673 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.491395+0000", 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40795189999999998 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(osd.11.21:1 3.0s0 3:0c3f65fa:::smithi18128752-8:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.088607+0000", 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.183354548000001, 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40355619300000001, 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.317 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.088607+0000", 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.088607+0000", 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.088609+0000", 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9470000000000002e-06 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.088629+0000", 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9434999999999999e-05 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.088649+0000", 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0449999999999999e-05 2024-04-26T01:24:19.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.088667+0000", 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7989e-05 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.286162+0000", 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19749450499999999 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.286344+0000", 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018260800000000001 2024-04-26T01:24:19.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.492163+0000", 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.205819259 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(osd.5.21:1 3.9s0 3:958f0a11:::smithi18128752-2:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.390739+0000", 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.881222466000001, 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40481407899999999, 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.321 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.390739+0000", 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.390739+0000", 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.390742+0000", 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4339999999999999e-06 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.390753+0000", 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1243000000000001e-05 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.390775+0000", 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1506e-05 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.390793+0000", 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8796000000000001e-05 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.492193+0000", 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101399925 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.492343+0000", 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014971299999999999 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.795553+0000", 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30321046200000001 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.325 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.326 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.326 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:310 3.9s0 3:958f0a11:::smithi18128752-2:head [read 0~180224,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:19.326 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.614273+0000", 2024-04-26T01:24:19.326 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.657688898, 2024-04-26T01:24:19.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.304100712, 2024-04-26T01:24:19.327 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.327 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.327 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.327 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.327 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.327 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.328 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 310 2024-04-26T01:24:19.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.328 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614273+0000", 2024-04-26T01:24:19.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614273+0000", 2024-04-26T01:24:19.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614276+0000", 2024-04-26T01:24:19.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.943e-06 2024-04-26T01:24:19.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.331 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614287+0000", 2024-04-26T01:24:19.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0753e-05 2024-04-26T01:24:19.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.331 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614317+0000", 2024-04-26T01:24:19.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0439000000000001e-05 2024-04-26T01:24:19.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614348+0000", 2024-04-26T01:24:19.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0719999999999997e-05 2024-04-26T01:24:19.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614481+0000", 2024-04-26T01:24:19.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013306700000000001 2024-04-26T01:24:19.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614597+0000", 2024-04-26T01:24:19.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011643500000000001 2024-04-26T01:24:19.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.335 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.335 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.918374+0000", 2024-04-26T01:24:19.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.303776355 2024-04-26T01:24:19.335 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.335 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.335 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.336 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:311 3.9s0 3:958f0a11:::smithi18128752-2:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:19.336 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.614383+0000", 2024-04-26T01:24:19.336 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.6575786940000001, 2024-04-26T01:24:19.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30518432400000001, 2024-04-26T01:24:19.336 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.337 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.337 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.337 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.337 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.337 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.337 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 311 2024-04-26T01:24:19.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.338 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614383+0000", 2024-04-26T01:24:19.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614383+0000", 2024-04-26T01:24:19.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614384+0000", 2024-04-26T01:24:19.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.02e-06 2024-04-26T01:24:19.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614390+0000", 2024-04-26T01:24:19.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4299999999999997e-06 2024-04-26T01:24:19.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.341 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.341 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614402+0000", 2024-04-26T01:24:19.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2046e-05 2024-04-26T01:24:19.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.341 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614422+0000", 2024-04-26T01:24:19.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0509999999999998e-05 2024-04-26T01:24:19.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715192+0000", 2024-04-26T01:24:19.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100769699 2024-04-26T01:24:19.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715469+0000", 2024-04-26T01:24:19.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027702 2024-04-26T01:24:19.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.919567+0000", 2024-04-26T01:24:19.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20409859899999999 2024-04-26T01:24:19.344 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.344 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.345 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.345 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:312 3.9s0 3:958f0a11:::smithi18128752-2:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:19.345 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.614475+0000", 2024-04-26T01:24:19.345 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.6574872520000001, 2024-04-26T01:24:19.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40588339800000001, 2024-04-26T01:24:19.346 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.346 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.346 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.346 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.347 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.347 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 312 2024-04-26T01:24:19.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.347 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.347 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.347 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614475+0000", 2024-04-26T01:24:19.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614475+0000", 2024-04-26T01:24:19.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614476+0000", 2024-04-26T01:24:19.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:24:19.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614482+0000", 2024-04-26T01:24:19.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2299999999999996e-06 2024-04-26T01:24:19.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614499+0000", 2024-04-26T01:24:19.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7028999999999999e-05 2024-04-26T01:24:19.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714580+0000", 2024-04-26T01:24:19.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10008127999999999 2024-04-26T01:24:19.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.716796+0000", 2024-04-26T01:24:19.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002215561 2024-04-26T01:24:19.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.716891+0000", 2024-04-26T01:24:19.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4915999999999997e-05 2024-04-26T01:24:19.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.020358+0000", 2024-04-26T01:24:19.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30346740900000002 2024-04-26T01:24:19.354 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.354 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.355 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.355 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:313 3.0s0 3:03a7f525:::smithi18128752-19:head [append 0~671744] snapc 6=[4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:19.355 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.714620+0000", 2024-04-26T01:24:19.356 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.5573421060000001, 2024-04-26T01:24:19.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51226638199999996, 2024-04-26T01:24:19.356 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.356 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.356 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.356 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.356 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.357 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.357 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 313 2024-04-26T01:24:19.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.357 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714620+0000", 2024-04-26T01:24:19.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714620+0000", 2024-04-26T01:24:19.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714622+0000", 2024-04-26T01:24:19.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.857e-06 2024-04-26T01:24:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715165+0000", 2024-04-26T01:24:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054386400000000002 2024-04-26T01:24:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715181+0000", 2024-04-26T01:24:19.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5094e-05 2024-04-26T01:24:19.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715191+0000", 2024-04-26T01:24:19.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0074999999999999e-05 2024-04-26T01:24:19.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.853370+0000", 2024-04-26T01:24:19.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13817900999999999 2024-04-26T01:24:19.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:19.363 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.854595+0000", 2024-04-26T01:24:19.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012252820000000001 2024-04-26T01:24:19.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.961048+0000", 2024-04-26T01:24:19.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.106452781 2024-04-26T01:24:19.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.961259+0000", 2024-04-26T01:24:19.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000210919 2024-04-26T01:24:19.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963480+0000", 2024-04-26T01:24:19.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022215469999999999 2024-04-26T01:24:19.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969408+0000", 2024-04-26T01:24:19.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0059276279999999999 2024-04-26T01:24:19.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.226288+0000", 2024-04-26T01:24:19.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25688034700000001 2024-04-26T01:24:19.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.368 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.368 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.226886+0000", 2024-04-26T01:24:19.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059797799999999999 2024-04-26T01:24:19.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.368 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.369 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:314 3.0s0 3:03a7f525:::smithi18128752-19:head [append 0~376832] snapc 6=[4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:19.369 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:19.369 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.556756246, 2024-04-26T01:24:19.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51249058199999997, 2024-04-26T01:24:19.369 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.370 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.370 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.370 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.370 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.370 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.370 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 314 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7300000000000005e-07 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715477+0000", 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027050499999999999 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715489+0000", 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2033e-05 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715497+0000", 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9799999999999998e-06 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.854744+0000", 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.139247491 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855021+0000", 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027672100000000001 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.965922+0000", 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.110901033 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.966294+0000", 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00037151900000000001 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968002+0000", 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001707914 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972283+0000", 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0042819019999999998 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.227207+0000", 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25492373400000001 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.227696+0000", 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00048897700000000003 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.375 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:315 3.0s0 3:03a7f525:::smithi18128752-19:head [setxattr _header (56) in=7b] snapc 6=[4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.5564511940000001, 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53997301399999997, 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 315 2024-04-26T01:24:19.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-04-26T01:24:19.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715515+0000", 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0879999999999997e-06 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715523+0000", 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.605e-06 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715529+0000", 2024-04-26T01:24:19.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1770000000000003e-06 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855140+0000", 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13961058700000001 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855264+0000", 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012369900000000001 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969747+0000", 2024-04-26T01:24:19.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.114483211 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969923+0000", 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000175954 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970982+0000", 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010588049999999999 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972408+0000", 2024-04-26T01:24:19.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014263259999999999 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.254671+0000", 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28226294200000002 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.255484+0000", 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081285000000000005 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:316 3.0s0 3:03a7f525:::smithi18128752-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.556424668, 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86208679600000004, 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 316 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715538+0000", 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.68e-07 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715540+0000", 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.413e-06 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715547+0000", 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.131e-06 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715553+0000", 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6640000000000003e-06 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855355+0000", 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13980195500000001 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855595+0000", 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000240265 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971741+0000", 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.116145744 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972105+0000", 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036422899999999999 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272430+0000", 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30032516799999998 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272617+0000", 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018708799999999999 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.577624+0000", 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305006571 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.9s0 46/30 ECSubReadReply(tid=66, attrs_read=0))", 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.717679+0000", 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.5542830410000001, 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30122311800000001, 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:19.387 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.717679+0000", 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.717679+0000", 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.717680+0000", 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.136e-06 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.717760+0000", 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0492000000000001e-05 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.717779+0000", 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8885999999999998e-05 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.817882+0000", 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100103054 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.018844+0000", 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20096165299999999 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.018902+0000", 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7896999999999998e-05 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.390 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:332 3.9s0 3:93dca175:::smithi18128752-10:head [stat,delete] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:18.474248+0000", 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.79771368600000003, 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40412863399999999, 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 332 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:19.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.474248+0000", 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.474248+0000", 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.474250+0000", 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7230000000000001e-06 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:19.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.474258+0000", 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9270000000000005e-06 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.474276+0000", 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8056000000000001e-05 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.474289+0000", 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3293e-05 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:19.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.574391+0000", 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10010229499999999 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.574466+0000", 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4344999999999994e-05 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.575016+0000", 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055062299999999998 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.575981+0000", 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00096422800000000003 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.878282+0000", 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30230111900000001 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.878377+0000", 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5025000000000006e-05 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.395 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.396 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:19.396 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:19.396 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:19.397 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:19.398 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:19.398 INFO:tasks.rados.rados.0.smithi181.stdout:132: writing smithi18128752-48 from 802816 to 1523712 tid 1 2024-04-26T01:24:19.398 INFO:tasks.rados.rados.0.smithi181.stdout:132: writing smithi18128752-48 from 1523712 to 1900544 tid 2 2024-04-26T01:24:19.398 INFO:tasks.rados.rados.0.smithi181.stdout:133: read oid 47 snap 2 2024-04-26T01:24:19.398 INFO:tasks.rados.rados.0.smithi181.stdout:133: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:24:19.398 INFO:tasks.rados.rados.0.smithi181.stdout:134: rollback oid 27 current snap is 6 2024-04-26T01:24:19.398 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 27 to 2 2024-04-26T01:24:19.398 INFO:tasks.rados.rados.0.smithi181.stdout:135: delete oid 17 current snap is 6 2024-04-26T01:24:19.399 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:19.399 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:19.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:19.400 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:19.396+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:19.420 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:19.420 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:19.498 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:19.500+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:19.530 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:19.530 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:19.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:19.564 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 23 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-04-26T01:24:19.592 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 1 to smithi18128752-48 2024-04-26T01:24:19.592 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 2 to smithi18128752-48 2024-04-26T01:24:19.593 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 3 to smithi18128752-48 2024-04-26T01:24:19.599 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:19.600+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:19.629 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:19.629 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:19.668 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:19.668 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:19.670 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:19.670 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:19.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:19.690 INFO:tasks.rados.rados.0.smithi181.stdout:131: done (4 left) 2024-04-26T01:24:19.690 INFO:tasks.rados.rados.0.smithi181.stdout:135: done (3 left) 2024-04-26T01:24:19.690 INFO:tasks.rados.rados.0.smithi181.stdout:136: delete oid 19 current snap is 6 2024-04-26T01:24:19.709 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:19.709 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:19.735 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:19.737+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:19.740 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:19.740 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:19.773 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:19.773 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:19.800 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:19.803+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:19.804 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:19.805 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:19.813 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:19.813 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:19.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:19.833 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:19.833 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:19.861 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:19.861 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:19.863 INFO:tasks.rados.rados.0.smithi181.stdout:134: finishing rollback tid 0 to smithi18128752-27 2024-04-26T01:24:19.890 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:19.890 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:19.893 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 4 to smithi18128752-48 2024-04-26T01:24:19.893 INFO:tasks.rados.rados.0.smithi181.stdout:132: oid 48 updating version 0 to 31 2024-04-26T01:24:19.893 INFO:tasks.rados.rados.0.smithi181.stdout:132: oid 48 updating version 31 to 33 2024-04-26T01:24:19.893 INFO:tasks.rados.rados.0.smithi181.stdout:132: oid 48 version 33 is already newer than 32 2024-04-26T01:24:19.893 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 33 (ObjNum 65 snap 6 seq_num 65) dirty exists 2024-04-26T01:24:19.893 INFO:tasks.rados.rados.0.smithi181.stdout:132: left oid 48 (ObjNum 65 snap 6 seq_num 65) 2024-04-26T01:24:19.893 INFO:tasks.rados.rados.0.smithi181.stdout:133: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:24:19.902 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:19.904+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:19.913 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:19.913 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:19.937 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:19.937 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:19.964 INFO:tasks.rados.rados.0.smithi181.stdout:134: finishing rollback tid 1 to smithi18128752-27 2024-04-26T01:24:19.965 INFO:tasks.rados.rados.0.smithi181.stdout:134: finishing rollback tid 2 to smithi18128752-27 2024-04-26T01:24:19.965 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 27 v 24 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-04-26T01:24:19.983 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:19.983 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:19.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:20.003 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:20.004+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:20.104 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:20.104+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:20.141 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15] out_osds: [7, 12, 2] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:24:20.141 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-26T01:24:20.164 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:20.164 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:20.179 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:24:20.223 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:20.225+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:20.274 INFO:tasks.rados.rados.0.smithi181.stdout:132: done (3 left) 2024-04-26T01:24:20.274 INFO:tasks.rados.rados.0.smithi181.stdout:133: done (2 left) 2024-04-26T01:24:20.274 INFO:tasks.rados.rados.0.smithi181.stdout:134: done (1 left) 2024-04-26T01:24:20.274 INFO:tasks.rados.rados.0.smithi181.stdout:136: done (0 left) 2024-04-26T01:24:20.274 INFO:tasks.rados.rados.0.smithi181.stdout:137: copy_from oid 36 from oid 1 current snap is 6 2024-04-26T01:24:20.274 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 31 current snap is 6 2024-04-26T01:24:20.274 INFO:tasks.rados.rados.0.smithi181.stdout:138: seq_num 68 ranges {507904=770048,1277952=376832} 2024-04-26T01:24:20.294 INFO:tasks.rados.rados.0.smithi181.stdout:138: writing smithi18128752-31 from 507904 to 1277952 tid 1 2024-04-26T01:24:20.299 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:20.299 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:20.301 INFO:tasks.rados.rados.0.smithi181.stdout:138: writing smithi18128752-31 from 1277952 to 1654784 tid 2 2024-04-26T01:24:20.301 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 8 current snap is 6 2024-04-26T01:24:20.301 INFO:tasks.rados.rados.0.smithi181.stdout:139: seq_num 69 ranges {131072=786432,917504=376832} 2024-04-26T01:24:20.306 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:20.308+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:20.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:24:20.316 INFO:tasks.rados.rados.0.smithi181.stdout:139: writing smithi18128752-8 from 131072 to 917504 tid 1 2024-04-26T01:24:20.319 INFO:tasks.rados.rados.0.smithi181.stdout:139: writing smithi18128752-8 from 917504 to 1294336 tid 2 2024-04-26T01:24:20.319 INFO:tasks.rados.rados.0.smithi181.stdout:140: copy_from oid 41 from oid 17 current snap is 6 2024-04-26T01:24:20.319 INFO:tasks.rados.rados.0.smithi181.stdout:141: copy_from oid 14 from oid 25 current snap is 6 2024-04-26T01:24:20.319 INFO:tasks.rados.rados.0.smithi181.stdout:142: snap_create 2024-04-26T01:24:20.407 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:20.408+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:20.440 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:20.440 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:20.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:24:20.507 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:20.509+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:20.521 INFO:tasks.rados.rados.0.smithi181.stdout:140: finishing copy_from to smithi18128752-41 2024-04-26T01:24:20.522 INFO:tasks.rados.rados.0.smithi181.stdout:140: got expected ENOENT (src dne) 2024-04-26T01:24:20.578 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:20.578 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:20.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:24:20.609 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:20.612+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:20.709 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:20.709+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:20.720 INFO:tasks.rados.rados.0.smithi181.stdout:140: finishing copy_from racing read to smithi18128752-41 2024-04-26T01:24:20.720 INFO:tasks.rados.rados.0.smithi181.stdout:137: finishing copy_from racing read to smithi18128752-36 2024-04-26T01:24:20.732 INFO:tasks.rados.rados.0.smithi181.stdout:139: finishing write tid 1 to smithi18128752-8 2024-04-26T01:24:20.765 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:20.765 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:20.780 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:24:20.810 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:20.812+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:20.821 INFO:tasks.rados.rados.0.smithi181.stdout:137: finishing copy_from to smithi18128752-36 2024-04-26T01:24:20.821 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 14 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-26T01:24:20.828 INFO:tasks.rados.rados.0.smithi181.stdout:141: finishing copy_from racing read to smithi18128752-14 2024-04-26T01:24:20.833 INFO:tasks.rados.rados.0.smithi181.stdout:139: finishing write tid 2 to smithi18128752-8 2024-04-26T01:24:20.911 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:20.913+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:20.935 INFO:tasks.rados.rados.0.smithi181.stdout:139: finishing write tid 3 to smithi18128752-8 2024-04-26T01:24:20.939 INFO:tasks.rados.rados.0.smithi181.stdout:138: finishing write tid 1 to smithi18128752-31 2024-04-26T01:24:20.944 INFO:tasks.rados.rados.0.smithi181.stdout:138: finishing write tid 2 to smithi18128752-31 2024-04-26T01:24:20.946 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:20.946 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:20.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:24:21.012 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:21.012+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:21.040 INFO:tasks.rados.rados.0.smithi181.stdout:139: finishing write tid 4 to smithi18128752-8 2024-04-26T01:24:21.040 INFO:tasks.rados.rados.0.smithi181.stdout:139: oid 8 updating version 0 to 28 2024-04-26T01:24:21.040 INFO:tasks.rados.rados.0.smithi181.stdout:139: oid 8 version 28 is already newer than 27 2024-04-26T01:24:21.040 INFO:tasks.rados.rados.0.smithi181.stdout:139: oid 8 updating version 28 to 29 2024-04-26T01:24:21.040 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 29 (ObjNum 69 snap 6 seq_num 69) dirty exists 2024-04-26T01:24:21.040 INFO:tasks.rados.rados.0.smithi181.stdout:139: left oid 8 (ObjNum 69 snap 6 seq_num 69) 2024-04-26T01:24:21.043 INFO:tasks.rados.rados.0.smithi181.stdout:138: finishing write tid 3 to smithi18128752-31 2024-04-26T01:24:21.113 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:21.115+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:21.125 INFO:tasks.rados.rados.0.smithi181.stdout:141: finishing copy_from to smithi18128752-14 2024-04-26T01:24:21.125 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 14 v 10 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-04-26T01:24:21.127 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:21.127 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:21.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:24:21.147 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T01:24:21.153 INFO:tasks.rados.rados.0.smithi181.stdout:137: done (5 left) 2024-04-26T01:24:21.153 INFO:tasks.rados.rados.0.smithi181.stdout:139: done (4 left) 2024-04-26T01:24:21.153 INFO:tasks.rados.rados.0.smithi181.stdout:140: done (3 left) 2024-04-26T01:24:21.153 INFO:tasks.rados.rados.0.smithi181.stdout:141: done (2 left) 2024-04-26T01:24:21.153 INFO:tasks.rados.rados.0.smithi181.stdout:142: done (1 left) 2024-04-26T01:24:21.153 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 14 current snap is 7 2024-04-26T01:24:21.153 INFO:tasks.rados.rados.0.smithi181.stdout:143: seq_num 70 ranges {491520=802816,1294336=376832} 2024-04-26T01:24:21.160 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:24:21.163 INFO:tasks.rados.rados.0.smithi181.stdout:143: writing smithi18128752-14 from 491520 to 1294336 tid 1 2024-04-26T01:24:21.167 INFO:tasks.rados.rados.0.smithi181.stdout:143: writing smithi18128752-14 from 1294336 to 1671168 tid 2 2024-04-26T01:24:21.167 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 24 current snap is 7 2024-04-26T01:24:21.167 INFO:tasks.rados.rados.0.smithi181.stdout:144: seq_num 71 ranges {1392640=425984,1818624=573440,2392064=196608} 2024-04-26T01:24:21.176 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-26T01:24:21.181 INFO:tasks.rados.rados.0.smithi181.stdout:144: writing smithi18128752-24 from 1392640 to 1818624 tid 1 2024-04-26T01:24:21.187 INFO:tasks.rados.rados.0.smithi181.stdout:144: writing smithi18128752-24 from 1818624 to 2392064 tid 2 2024-04-26T01:24:21.189 INFO:tasks.rados.rados.0.smithi181.stdout:144: writing smithi18128752-24 from 2392064 to 2588672 tid 3 2024-04-26T01:24:21.189 INFO:tasks.rados.rados.0.smithi181.stdout:145: rollback oid 3 current snap is 7 2024-04-26T01:24:21.189 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 3 to 2 2024-04-26T01:24:21.189 INFO:tasks.rados.rados.0.smithi181.stdout:146: snap_create 2024-04-26T01:24:21.219 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:21.223+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:21.280 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:21.280 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:21.294 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:24:21.315 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:21.316+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:21.416 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:21.416 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:21.416 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:21.419+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:21.431 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:24:21.518 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:21.520+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:21.535 INFO:tasks.rados.rados.0.smithi181.stdout:145: finishing rollback tid 0 to smithi18128752-3 2024-04-26T01:24:21.535 INFO:tasks.rados.rados.0.smithi181.stdout:145: finishing rollback tid 1 to smithi18128752-3 2024-04-26T01:24:21.600 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:21.601 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:21.615 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:24:21.618 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:21.619+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:21.649 INFO:tasks.rados.rados.0.smithi181.stdout:145: finishing rollback tid 2 to smithi18128752-3 2024-04-26T01:24:21.649 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 23 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-26T01:24:21.669 INFO:tasks.rados.rados.0.smithi181.stdout:143: finishing write tid 1 to smithi18128752-14 2024-04-26T01:24:21.669 INFO:tasks.rados.rados.0.smithi181.stdout:143: finishing write tid 2 to smithi18128752-14 2024-04-26T01:24:21.719 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:21.720+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:21.739 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:21.739 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:21.749 INFO:tasks.rados.rados.0.smithi181.stdout:138: finishing write tid 4 to smithi18128752-31 2024-04-26T01:24:21.749 INFO:tasks.rados.rados.0.smithi181.stdout:138: oid 31 updating version 0 to 39 2024-04-26T01:24:21.750 INFO:tasks.rados.rados.0.smithi181.stdout:138: oid 31 version 39 is already newer than 38 2024-04-26T01:24:21.750 INFO:tasks.rados.rados.0.smithi181.stdout:138: oid 31 version 39 is already newer than 37 2024-04-26T01:24:21.750 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 39 (ObjNum 68 snap 6 seq_num 68) dirty exists 2024-04-26T01:24:21.750 INFO:tasks.rados.rados.0.smithi181.stdout:138: left oid 31 (ObjNum 68 snap 6 seq_num 68) 2024-04-26T01:24:21.753 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:24:21.771 INFO:tasks.rados.rados.0.smithi181.stdout:143: finishing write tid 3 to smithi18128752-14 2024-04-26T01:24:21.821 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:21.825+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:21.873 INFO:tasks.rados.rados.0.smithi181.stdout:143: finishing write tid 4 to smithi18128752-14 2024-04-26T01:24:21.874 INFO:tasks.rados.rados.0.smithi181.stdout:143: oid 14 updating version 0 to 31 2024-04-26T01:24:21.874 INFO:tasks.rados.rados.0.smithi181.stdout:143: oid 14 version 31 is already newer than 29 2024-04-26T01:24:21.874 INFO:tasks.rados.rados.0.smithi181.stdout:143: oid 14 version 31 is already newer than 30 2024-04-26T01:24:21.874 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 14 v 31 (ObjNum 70 snap 7 seq_num 70) dirty exists 2024-04-26T01:24:21.874 INFO:tasks.rados.rados.0.smithi181.stdout:143: left oid 14 (ObjNum 70 snap 7 seq_num 70) 2024-04-26T01:24:21.882 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:21.882 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:21.894 INFO:tasks.rados.rados.0.smithi181.stdout:144: finishing write tid 1 to smithi18128752-24 2024-04-26T01:24:21.897 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:24:21.922 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:21.924+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:21.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:21.995 INFO:tasks.rados.rados.0.smithi181.stdout:144: finishing write tid 2 to smithi18128752-24 2024-04-26T01:24:21.995 INFO:tasks.rados.rados.0.smithi181.stdout:144: finishing write tid 3 to smithi18128752-24 2024-04-26T01:24:22.022 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:22.024+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:22.024 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:22.024 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:22.040 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:24:22.109 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:22.110 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:22.123 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:22.127+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:22.131 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:22.131 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:22.152 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:22.152 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:22.160 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-04-26T01:24:22.160 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T01:24:22.169 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:22.169 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:138: done (4 left) 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:143: done (3 left) 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:145: done (2 left) 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:146: done (1 left) 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:147: read oid 33 snap -1 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:147: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:148: read oid 20 snap -1 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:148: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:22.170 INFO:tasks.rados.rados.0.smithi181.stdout:149: snap_create 2024-04-26T01:24:22.177 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:22.177 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:22.195 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:24:22.197 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-26T01:24:22.239 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:22.243+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:22.292 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:22.292 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:22.313 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:22.313 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:22.325 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:22.325+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:22.339 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:22.340 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:22.357 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:22.365 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:22.365 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:22.378 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:24:22.385 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:22.385 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:22.400 INFO:tasks.rados.rados.0.smithi181.stdout:144: finishing write tid 4 to smithi18128752-24 2024-04-26T01:24:22.406 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:22.406 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:22.425 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:22.427+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:22.427 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:22.428 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:22.448 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:22.449 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:22.465 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:22.466 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:22.477 INFO:tasks.rados.rados.0.smithi181.stdout:147: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:22.488 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:22.488 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:22.502 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:22.502 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:22.502 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:22.502 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:22.510 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:22.510 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:22.515 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:24:22.532 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:22.532 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:22.532 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:22.535+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:22.576 INFO:tasks.rados.rados.0.smithi181.stdout:144: finishing write tid 5 to smithi18128752-24 2024-04-26T01:24:22.577 INFO:tasks.rados.rados.0.smithi181.stdout:144: oid 24 updating version 0 to 25 2024-04-26T01:24:22.577 INFO:tasks.rados.rados.0.smithi181.stdout:144: oid 24 updating version 25 to 27 2024-04-26T01:24:22.577 INFO:tasks.rados.rados.0.smithi181.stdout:144: oid 24 updating version 27 to 28 2024-04-26T01:24:22.577 INFO:tasks.rados.rados.0.smithi181.stdout:144: oid 24 version 28 is already newer than 26 2024-04-26T01:24:22.577 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 28 (ObjNum 71 snap 7 seq_num 71) dirty exists 2024-04-26T01:24:22.577 INFO:tasks.rados.rados.0.smithi181.stdout:144: left oid 24 (ObjNum 71 snap 7 seq_num 71) 2024-04-26T01:24:22.577 INFO:tasks.rados.rados.0.smithi181.stdout:148: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:22.628 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:22.628+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=13, reqid=client.4589.0:74, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.592886183000001, 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20101313800000001, 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.645 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054193+0000", 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.13e-06 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054208+0000", 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4562000000000001e-05 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054223+0000", 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5536000000000002e-05 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.647 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.154317+0000", 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100094023 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.254494+0000", 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100176788 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.254508+0000", 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3779e-05 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.648 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255170+0000", 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00066210200000000002 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255205+0000", 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5218000000000001e-05 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.649 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=16, reqid=client.4589.0:86, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.460066775000001, 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.1988666, 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.650 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187013+0000", 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8840000000000001e-06 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187073+0000", 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9972000000000001e-05 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.651 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187121+0000", 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7222999999999998e-05 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187151+0000", 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0107000000000002e-05 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284476+0000", 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.097325196000000003 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.652 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284487+0000", 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1633e-05 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385837+0000", 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10134913199999999 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385878+0000", 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1452999999999998e-05 2024-04-26T01:24:22.653 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=17, reqid=client.4589.0:146, at_version=35'9, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.456699391000001, 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19552889900000001, 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.654 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190380+0000", 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5099999999999999e-06 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.655 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190583+0000", 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020251199999999999 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190609+0000", 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6256999999999999e-05 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190624+0000", 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5302999999999999e-05 2024-04-26T01:24:22.656 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.384789+0000", 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.194164533 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.384800+0000", 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0919e-05 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385883+0000", 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001082902 2024-04-26T01:24:22.657 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385908+0000", 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4963000000000001e-05 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.453894833, 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19307022800000001, 2024-04-26T01:24:22.658 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193185+0000", 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193366+0000", 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000181096 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193393+0000", 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6673e-05 2024-04-26T01:24:22.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193408+0000", 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.554e-05 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385391+0000", 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19198310199999999 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385402+0000", 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0586e-05 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.386222+0000", 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081973500000000004 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.386254+0000", 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1866000000000003e-05 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.662 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=19, reqid=client.4589.0:148, at_version=35'11, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.496969+0000", 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.150109633, 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20146192700000001, 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.496969+0000", 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.496969+0000", 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.496970+0000", 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8029999999999999e-06 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497037+0000", 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6655999999999993e-05 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497066+0000", 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8663e-05 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497084+0000", 2024-04-26T01:24:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8430000000000001e-05 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.697353+0000", 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20026850900000001 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.697368+0000", 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4970000000000001e-05 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.698370+0000", 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010020070000000001 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.698431+0000", 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0888999999999999e-05 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:185 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:22.667 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.57386706, 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.395059522, 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 185 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073214+0000", 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073225+0000", 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0404999999999999e-05 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073253+0000", 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8058000000000001e-05 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173330+0000", 2024-04-26T01:24:22.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10007712000000001 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.261568+0000", 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.088237503999999994 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.261653+0000", 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5259999999999993e-05 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.262369+0000", 2024-04-26T01:24:22.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00071622000000000005 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265729+0000", 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003360264 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.468152+0000", 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20242310199999999 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.468271+0000", 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000118279 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(osd.4.21:2 3.4s0 3:2e10a639:::smithi18128752-28:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.486989619999999, 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30734204100000001, 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.673 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160090+0000", 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7129999999999999e-06 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160104+0000", 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3596e-05 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160114+0000", 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8190000000000003e-06 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160125+0000", 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1413999999999999e-05 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160185+0000", 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9914000000000003e-05 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160251+0000", 2024-04-26T01:24:22.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5728999999999994e-05 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467431+0000", 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30717985599999997 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:186 3.4s0 3:2be1e129:::smithi18128752-30:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.473730945, 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39528098299999997, 2024-04-26T01:24:22.677 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 186 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173348+0000", 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.108e-06 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173354+0000", 2024-04-26T01:24:22.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3009999999999997e-06 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173362+0000", 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9710000000000004e-06 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173367+0000", 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3179999999999997e-06 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263197+0000", 2024-04-26T01:24:22.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.089829862999999996 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263286+0000", 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9519999999999997e-05 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263786+0000", 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00050013300000000004 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265873+0000", 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002086606 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568417+0000", 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30254369399999997 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568628+0000", 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021146899999999999 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:187 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.473703727, 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51780501899999998, 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.683 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 187 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173375+0000", 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1699999999999999e-07 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173377+0000", 2024-04-26T01:24:22.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.939e-06 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173381+0000", 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2050000000000004e-06 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173384+0000", 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2760000000000001e-06 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264258+0000", 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.090873535000000005 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264532+0000", 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027448400000000002 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265351+0000", 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081914900000000003 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.466871+0000", 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20151915000000001 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690961+0000", 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22409068500000001 2024-04-26T01:24:22.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.691180+0000", 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021827899999999999 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 37/30 ECSubWriteReply(tid=14, last_complete=37'9, committed=1, applied=1))", 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:22.689 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.382398033999999, 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.203106276, 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264681+0000", 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5999999999999991e-07 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264687+0000", 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9549999999999997e-06 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264702+0000", 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.486e-05 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.364796+0000", 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100093734 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467752+0000", 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102956666 2024-04-26T01:24:22.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467786+0000", 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4100999999999998e-05 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 37/30 ECSubWriteReply(tid=14, last_complete=37'9, committed=1, applied=1))", 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.389027+0000", 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.258050967999999, 2024-04-26T01:24:22.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.179630282, 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.389027+0000", 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.389027+0000", 2024-04-26T01:24:22.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.389028+0000", 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9999999999999995e-07 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.389036+0000", 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.2629999999999997e-06 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.389062+0000", 2024-04-26T01:24:22.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6227000000000001e-05 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.389074+0000", 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.208e-05 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568170+0000", 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17909578500000001 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568658+0000", 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00048792699999999998 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 37/30 ECSubWriteReply(tid=15, last_complete=37'10, committed=1, applied=1))", 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.490328+0000", 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.156749769999999, 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20036198099999999, 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490328+0000", 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490328+0000", 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490330+0000", 2024-04-26T01:24:22.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9470000000000002e-06 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490340+0000", 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6870000000000006e-06 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490370+0000", 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9952000000000001e-05 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.590485+0000", 2024-04-26T01:24:22.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10011507 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690653+0000", 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100167637 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690690+0000", 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7688000000000002e-05 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 40/34 ECSubWrite(tid=40, reqid=client.4589.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=40'23))", 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.008805+0000", 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.638272860000001, 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18152664700000001, 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008805+0000", 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008805+0000", 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008807+0000", 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.734e-06 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008833+0000", 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5772999999999999e-05 2024-04-26T01:24:22.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008862+0000", 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9003e-05 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008879+0000", 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6699999999999999e-05 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.089652+0000", 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.080773257000000001 2024-04-26T01:24:22.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.089677+0000", 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5185999999999999e-05 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.190283+0000", 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100606271 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.190332+0000", 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8723000000000001e-05 2024-04-26T01:24:22.704 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 40/30 ECSubWriteReply(tid=20, last_complete=41'12, committed=1, applied=1))", 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.093963+0000", 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.553114734999999, 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20213109300000001, 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093963+0000", 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093963+0000", 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093966+0000", 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2000000000000001e-06 2024-04-26T01:24:22.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093977+0000", 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1226000000000001e-05 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.094000+0000", 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3516999999999999e-05 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.194129+0000", 2024-04-26T01:24:22.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100128113 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296007+0000", 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101878477 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296095+0000", 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7559999999999995e-05 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.708 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:270 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.474164735, 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22773885299999999, 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 270 2024-04-26T01:24:22.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172916+0000", 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0170000000000001e-06 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172932+0000", 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5766000000000001e-05 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172963+0000", 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0440999999999999e-05 2024-04-26T01:24:22.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172980+0000", 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7224e-05 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.295355+0000", 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.122375555 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.295455+0000", 2024-04-26T01:24:22.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9192999999999997e-05 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.400652+0000", 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.105197657 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:271 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:22.713 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.474062264000001, 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53167738200000003, 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 271 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173017+0000", 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1100000000000005e-07 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173022+0000", 2024-04-26T01:24:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4990000000000002e-06 2024-04-26T01:24:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173034+0000", 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1667e-05 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173044+0000", 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9070000000000002e-06 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398177+0000", 2024-04-26T01:24:22.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225132687 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398388+0000", 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000211325 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.704693+0000", 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30630548600000002 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:272 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.474020585, 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53273070499999997, 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 272 2024-04-26T01:24:22.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9200000000000001e-07 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173079+0000", 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1137000000000001e-05 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173093+0000", 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3198e-05 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173102+0000", 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1859999999999998e-06 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401582+0000", 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22848043000000001 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401813+0000", 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023127900000000001 2024-04-26T01:24:22.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.705788+0000", 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30397488299999997 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 41/34 ECSubWrite(tid=33, reqid=osd.11.0:33, at_version=41'13, trim_to=0'0, roll_forward_to=37'11))", 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.206421+0000", 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.440656911, 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19296364399999999, 2024-04-26T01:24:22.724 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206421+0000", 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206421+0000", 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206422+0000", 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3e-07 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206517+0000", 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4803999999999996e-05 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206549+0000", 2024-04-26T01:24:22.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2456000000000002e-05 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206574+0000", 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4453000000000001e-05 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296181+0000", 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.089607645 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296201+0000", 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9967999999999999e-05 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.399316+0000", 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103114385 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.399385+0000", 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9302999999999997e-05 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.728 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.4s0 41/30 ECSubRead(tid=24, to_read={3:2e10a639:::smithi18128752-28:head=0,114688,0}, subchunks={3:2e10a639:::smithi18128752-28:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.244753553000001, 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20080430399999999, 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402377+0000", 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2645999999999997e-05 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.502485+0000", 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100107906 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.602850+0000", 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100364604 2024-04-26T01:24:22.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.603129+0000", 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000279148 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/30 ECSubWrite(tid=126, reqid=client.4589.0:327, at_version=47'30, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.462580+0000", 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.1844983329999996, 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201538619, 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:22.732 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462580+0000", 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462580+0000", 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:22.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462582+0000", 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6640000000000001e-06 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462626+0000", 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4576000000000001e-05 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462667+0000", 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0769000000000002e-05 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462690+0000", 2024-04-26T01:24:22.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2900000000000001e-05 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.663034+0000", 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20034379699999999 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.663052+0000", 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8717000000000001e-05 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.664050+0000", 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00099748999999999997 2024-04-26T01:24:22.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.664118+0000", 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8706000000000003e-05 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:22.736 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:22.738 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:24:22.738 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:22.740+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:22.839 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:22.843+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:22.868 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:22.868 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:22.868 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:22.883 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:24:22.940 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:22.945+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:23.005 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:23.006 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:23.006 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:23.019 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:24:23.041 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:23.041+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.6s4 35/34 ECSubRead(tid=8, to_read={3:60f0095a:::smithi18128752-3:head=0,262144,0}, subchunks={3:60f0095a:::smithi18128752-3:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:23.130 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.714999993999999, 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100734536, 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:23.131 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420052+0000", 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.846e-06 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420069+0000", 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7493e-05 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420098+0000", 2024-04-26T01:24:23.132 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9079000000000001e-05 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420118+0000", 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9468999999999999e-05 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520369+0000", 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100251189 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520785+0000", 2024-04-26T01:24:23.133 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00041545999999999998 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.es6 36/34 ECSubRead(tid=24, to_read={3:738de3f3:::smithi18128752-9:head=0,262144,0}, subchunks={3:738de3f3:::smithi18128752-9:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.415308294000001, 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00074145500000000002, 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:23.134 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:23.135 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.03e-07 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719756+0000", 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4053000000000001e-05 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719789+0000", 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2913999999999997e-05 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.136 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719810+0000", 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0513000000000002e-05 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719965+0000", 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015534199999999999 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.720483+0000", 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051822999999999995 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:23.137 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:23.138 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:23.138 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:23.138 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:23.142 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:23.144+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:23.144 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:24:23.168 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T01:24:23.174 INFO:tasks.rados.rados.0.smithi181.stdout:144: done (3 left) 2024-04-26T01:24:23.174 INFO:tasks.rados.rados.0.smithi181.stdout:147: done (2 left) 2024-04-26T01:24:23.174 INFO:tasks.rados.rados.0.smithi181.stdout:148: done (1 left) 2024-04-26T01:24:23.174 INFO:tasks.rados.rados.0.smithi181.stdout:149: done (0 left) 2024-04-26T01:24:23.175 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 31 current snap is 9 2024-04-26T01:24:23.175 INFO:tasks.rados.rados.0.smithi181.stdout:150: seq_num 72 ranges {1654784=442368,2097152=720896,2818048=49152} 2024-04-26T01:24:23.186 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-26T01:24:23.203 INFO:tasks.rados.rados.0.smithi181.stdout:150: writing smithi18128752-31 from 1654784 to 2097152 tid 1 2024-04-26T01:24:23.214 INFO:tasks.rados.rados.0.smithi181.stdout:150: writing smithi18128752-31 from 2097152 to 2818048 tid 2 2024-04-26T01:24:23.215 INFO:tasks.rados.rados.0.smithi181.stdout:150: writing smithi18128752-31 from 2818048 to 2867200 tid 3 2024-04-26T01:24:23.215 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 16 current snap is 9 2024-04-26T01:24:23.215 INFO:tasks.rados.rados.0.smithi181.stdout:151: seq_num 73 ranges {262144=458752,720896=475136,1196032=294912} 2024-04-26T01:24:23.225 INFO:tasks.rados.rados.0.smithi181.stdout:151: writing smithi18128752-16 from 262144 to 720896 tid 1 2024-04-26T01:24:23.230 INFO:tasks.rados.rados.0.smithi181.stdout:151: writing smithi18128752-16 from 720896 to 1196032 tid 2 2024-04-26T01:24:23.234 INFO:tasks.rados.rados.0.smithi181.stdout:151: writing smithi18128752-16 from 1196032 to 1490944 tid 3 2024-04-26T01:24:23.234 INFO:tasks.rados.rados.0.smithi181.stdout:152: copy_from oid 10 from oid 39 current snap is 9 2024-04-26T01:24:23.234 INFO:tasks.rados.rados.0.smithi181.stdout:153: read oid 48 snap 2 2024-04-26T01:24:23.234 INFO:tasks.rados.rados.0.smithi181.stdout:153: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T01:24:23.234 INFO:tasks.rados.rados.0.smithi181.stdout:154: copy_from oid 50 from oid 1 current snap is 9 2024-04-26T01:24:23.234 INFO:tasks.rados.rados.0.smithi181.stdout:155: snap_remove snap 8 2024-04-26T01:24:23.241 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:23.244+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:23.282 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:23.282 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:23.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.282 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=67, reqid=client.4589.0:382, at_version=51'41, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.215125+0000", 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.072787315000000005, 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.072807585999999994, 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.215125+0000", 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.215125+0000", 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.215126+0000", 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0729999999999999e-06 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.215340+0000", 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021383400000000001 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.215366+0000", 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6565999999999999e-05 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.215383+0000", 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6807999999999999e-05 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 1 2024-04-26T01:24:23.285 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:23.297 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:24:23.342 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:23.341+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:23.443 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:23.444+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:23.468 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:23.468 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:23.468 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:23.468 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:23.468 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:23.482 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:24:23.544 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:23.547+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:23.645 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:23.647+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:23.665 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=11, reqid=client.4589.0:127, at_version=35'7, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.092164776000001, 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202990695, 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.666 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577859+0000", 2024-04-26T01:24:23.667 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.013e-06 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578257+0000", 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039834700000000001 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578289+0000", 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2016e-05 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578306+0000", 2024-04-26T01:24:23.668 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6460000000000002e-05 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678641+0000", 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100335199 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678656+0000", 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5233e-05 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780779+0000", 2024-04-26T01:24:23.669 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102122955 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780847+0000", 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8472000000000004e-05 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=12, reqid=client.4589.0:128, at_version=35'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.581325+0000", 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.088696369000001, 2024-04-26T01:24:23.670 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.199590883, 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581325+0000", 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581325+0000", 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.671 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581327+0000", 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9470000000000002e-06 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581393+0000", 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6187999999999997e-05 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.672 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581434+0000", 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1041999999999998e-05 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.581493+0000", 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9039000000000002e-05 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779641+0000", 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19814733200000001 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.673 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779656+0000", 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.571e-05 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780882+0000", 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012260389999999999 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780916+0000", 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3586e-05 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.674 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=29, reqid=client.4589.0:118, at_version=35'11, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.631448+0000", 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.038573233000001, 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15008649800000001, 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.675 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631448+0000", 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631448+0000", 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631451+0000", 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.683e-06 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.676 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631749+0000", 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000298246 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631785+0000", 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5709e-05 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631812+0000", 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6602999999999999e-05 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.677 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779726+0000", 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.147914885 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.779743+0000", 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6056999999999999e-05 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.781482+0000", 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001739167 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.678 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.781535+0000", 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3148e-05 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=30, reqid=client.4589.0:119, at_version=35'12, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.631852+0000", 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.038169737, 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15076103499999999, 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.679 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631852+0000", 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631852+0000", 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631853+0000", 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1999999999999999e-06 2024-04-26T01:24:23.680 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631890+0000", 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6813000000000001e-05 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631911+0000", 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1492999999999999e-05 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.631930+0000", 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9162000000000001e-05 2024-04-26T01:24:23.681 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780836+0000", 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.148905065 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780849+0000", 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3856000000000001e-05 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782457+0000", 2024-04-26T01:24:23.682 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016076300000000001 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782613+0000", 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000155816 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=31, reqid=client.4589.0:124, at_version=35'13, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.818643522999999, 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20482444299999999, 2024-04-26T01:24:23.683 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.684 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851380+0000", 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.973e-06 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852052+0000", 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067198400000000004 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852084+0000", 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2298999999999997e-05 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.685 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952200+0000", 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100116018 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952352+0000", 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015207299999999999 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952377+0000", 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4363e-05 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.686 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056116+0000", 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103738862 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056202+0000", 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6871000000000002e-05 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.687 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 35/30 ECSubWrite(tid=170, reqid=client.4589.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.700707035000001, 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20178349000000001, 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.688 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969317+0000", 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0600000000000002e-06 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969365+0000", 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8296e-05 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.689 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969439+0000", 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3907999999999998e-05 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969466+0000", 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7223e-05 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169873+0000", 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20040670999999999 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.690 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169887+0000", 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4727e-05 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171018+0000", 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001130968 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171098+0000", 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9598000000000006e-05 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.691 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.392936249000002, 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20597036399999999, 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:23.692 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277086+0000", 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.693 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277250+0000", 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016446899999999999 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277265+0000", 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.418e-05 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277275+0000", 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0456e-05 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.694 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481015+0000", 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203739686 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481042+0000", 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6962999999999999e-05 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482955+0000", 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001913816 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483056+0000", 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100204 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=15, reqid=client.4589.0:142, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.277533+0000", 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.392488243999999, 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.181090423, 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.696 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277533+0000", 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277533+0000", 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277534+0000", 2024-04-26T01:24:23.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8000000000000004e-07 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277731+0000", 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019659199999999999 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277754+0000", 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3612999999999999e-05 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277766+0000", 2024-04-26T01:24:23.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1839e-05 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.378073+0000", 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100307204 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.378088+0000", 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4673e-05 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.458572+0000", 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080483671000000007 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.458624+0000", 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1950999999999998e-05 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=16, reqid=client.4589.0:143, at_version=35'10, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.285657+0000", 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.384364701999999, 2024-04-26T01:24:23.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.173905958, 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285657+0000", 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285657+0000", 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285659+0000", 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0159999999999998e-06 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285899+0000", 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024038099999999999 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285943+0000", 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3849e-05 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.386077+0000", 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100133608 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.457796+0000", 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.071719632000000005 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.457812+0000", 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5982999999999999e-05 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.459449+0000", 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016369519999999999 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.459563+0000", 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011353699999999999 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=37, reqid=client.4589.0:134, at_version=0'0, trim_to=0'0, roll_forward_to=35'16))", 2024-04-26T01:24:23.704 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.380246+0000", 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.289775410000001, 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.178392667, 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380246+0000", 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380246+0000", 2024-04-26T01:24:23.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380248+0000", 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.72e-06 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380271+0000", 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3149000000000001e-05 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380301+0000", 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9952999999999999e-05 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.380319+0000", 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8312999999999999e-05 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.557548+0000", 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17722879 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.557566+0000", 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8016000000000002e-05 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.558575+0000", 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001008421 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.558639+0000", 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4305000000000002e-05 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.4s3 41/30 ECSubWrite(tid=22, reqid=osd.8.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.273469573, 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20184675299999999, 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396553+0000", 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.31e-06 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396576+0000", 2024-04-26T01:24:23.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2816e-05 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396597+0000", 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1166999999999999e-05 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396611+0000", 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4099e-05 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.496984+0000", 2024-04-26T01:24:23.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100372218 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.497002+0000", 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8051999999999999e-05 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.598348+0000", 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10134613200000001 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.598399+0000", 2024-04-26T01:24:23.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0958999999999998e-05 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.4s3 41/30 ECSubRead(tid=23, to_read={3:2e10a639:::smithi18128752-28:head=0,114688,0}, subchunks={3:2e10a639:::smithi18128752-28:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.399398+0000", 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.270623838000001, 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19836884900000001, 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.399398+0000", 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.399398+0000", 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.399399+0000", 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4529999999999999e-06 2024-04-26T01:24:23.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.399415+0000", 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5945999999999999e-05 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.399445+0000", 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9439999999999999e-05 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.399462+0000", 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7169000000000001e-05 2024-04-26T01:24:23.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597589+0000", 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.198127041 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597767+0000", 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017780000000000001 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.716 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 41/34 ECSubWrite(tid=45, reqid=osd.2.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.597646+0000", 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.072375190000001, 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20104497700000001, 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597646+0000", 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597646+0000", 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597649+0000", 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1799999999999999e-06 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597677+0000", 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8368999999999999e-05 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597716+0000", 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8655999999999997e-05 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597754+0000", 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8046000000000001e-05 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.698034+0000", 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100280385 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.698048+0000", 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4076999999999999e-05 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.798604+0000", 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10055604999999999 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.798691+0000", 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7213999999999995e-05 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.720 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 41/34 ECSubWrite(tid=46, reqid=osd.2.0:46, at_version=41'29, trim_to=0'0, roll_forward_to=41'27))", 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.599708+0000", 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.070313422, 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19902400200000001, 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599708+0000", 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599708+0000", 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599710+0000", 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5799999999999999e-06 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599748+0000", 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8299e-05 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599774+0000", 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6363000000000002e-05 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599795+0000", 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1058999999999999e-05 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.700010+0000", 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100214567 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.700024+0000", 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3847e-05 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.798699+0000", 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098675423999999998 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.798732+0000", 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2863e-05 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s2 41/34 ECSubWrite(tid=28, reqid=osd.0.0:27, at_version=0'0, trim_to=0'0, roll_forward_to=41'17))", 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.607387+0000", 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.062634452999999, 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19448689999999999, 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.725 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607387+0000", 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607387+0000", 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607388+0000", 2024-04-26T01:24:23.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4600000000000003e-07 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607410+0000", 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1804e-05 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607434+0000", 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4434999999999999e-05 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607460+0000", 2024-04-26T01:24:23.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5999999999999998e-05 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800701+0000", 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193240895 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800730+0000", 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8986e-05 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.801784+0000", 2024-04-26T01:24:23.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001054283 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.801874+0000", 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9650999999999999e-05 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/44 ECSubWrite(tid=71, reqid=client.4589.0:315, at_version=0'0, trim_to=0'0, roll_forward_to=46'20))", 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.4145873089999998, 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30303612499999999, 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.729 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255436+0000", 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.81e-06 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255469+0000", 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2895999999999998e-05 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255499+0000", 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0253e-05 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.355601+0000", 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100101499 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455871+0000", 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10027049 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455899+0000", 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7636000000000001e-05 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558401+0000", 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102502418 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558470+0000", 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9122999999999995e-05 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPush(3.8s2 47/44 [PushOp(3:1fe6984d:::smithi18128752-48:head, version: 35'11, data_included: [0~200704], data_size: 200704, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:1fe6984d:::smithi18128752-48:head@35'11, size: 802816, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 802816, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.360764+0000", 2024-04-26T01:24:23.733 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.3092573930000002, 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19762925000000001, 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.360764+0000", 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.360764+0000", 2024-04-26T01:24:23.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.360766+0000", 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6929999999999999e-06 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.361045+0000", 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027926699999999998 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.361090+0000", 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4456000000000003e-05 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.361119+0000", 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9312999999999999e-05 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.557251+0000", 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.196131634 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558393+0000", 2024-04-26T01:24:23.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001142887 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 47/44 ECSubWrite(tid=50, reqid=client.4589.0:324, at_version=0'0, trim_to=0'0, roll_forward_to=47'27))", 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.362166+0000", 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.3078553809999995, 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19639446799999999, 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.737 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.362166+0000", 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.362166+0000", 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.362168+0000", 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9829999999999999e-06 2024-04-26T01:24:23.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.362212+0000", 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3538999999999998e-05 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.362252+0000", 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9818999999999999e-05 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.362275+0000", 2024-04-26T01:24:23.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3876999999999999e-05 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.456589+0000", 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.094313544999999999 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.456604+0000", 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4803e-05 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558476+0000", 2024-04-26T01:24:23.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101871817 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558561+0000", 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5085000000000003e-05 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.bs2 47/34 ECSubWrite(tid=127, reqid=client.4589.0:331, at_version=0'0, trim_to=0'0, roll_forward_to=47'8))", 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.559137+0000", 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.1108842780000003, 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201584918, 2024-04-26T01:24:23.741 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559137+0000", 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559137+0000", 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559139+0000", 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.584e-06 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559183+0000", 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3764999999999999e-05 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559207+0000", 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3915999999999999e-05 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559222+0000", 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5866999999999999e-05 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.759571+0000", 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200348216 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.759586+0000", 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5255999999999999e-05 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.760642+0000", 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010563320000000001 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.760722+0000", 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9981999999999994e-05 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.745 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s3 47/44 ECSubRead(tid=53, to_read={3:18914e2a:::smithi18128752-43:head=262144,262144,0}, subchunks={3:18914e2a:::smithi18128752-43:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.670006+0000", 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.0000151930000003, 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20097707400000001, 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.670006+0000", 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.670006+0000", 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.670008+0000", 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9199999999999998e-06 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.670025+0000", 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6586e-05 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.670055+0000", 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0602999999999998e-05 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.770152+0000", 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10009688 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.870435+0000", 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100282856 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.870983+0000", 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00054822900000000001 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.749 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:23.751 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:24:23.751 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:23.752+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:23.763 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T01:24:23.770 INFO:tasks.rados.rados.0.smithi181.stdout:155: done (5 left) 2024-04-26T01:24:23.770 INFO:tasks.rados.rados.0.smithi181.stdout:156: delete oid 18 current snap is 9 2024-04-26T01:24:23.770 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 1 to smithi18128752-31 2024-04-26T01:24:23.770 INFO:tasks.rados.rados.0.smithi181.stdout:152: finishing copy_from racing read to smithi18128752-10 2024-04-26T01:24:23.770 INFO:tasks.rados.rados.0.smithi181.stdout:154: finishing copy_from racing read to smithi18128752-50 2024-04-26T01:24:23.770 INFO:tasks.rados.rados.0.smithi181.stdout:152: finishing copy_from to smithi18128752-10 2024-04-26T01:24:23.770 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 11 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-04-26T01:24:23.771 INFO:tasks.rados.rados.0.smithi181.stdout:151: finishing write tid 1 to smithi18128752-16 2024-04-26T01:24:23.771 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 2 to smithi18128752-31 2024-04-26T01:24:23.771 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 3 to smithi18128752-31 2024-04-26T01:24:23.771 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 4 to smithi18128752-31 2024-04-26T01:24:23.771 INFO:tasks.rados.rados.0.smithi181.stdout:151: finishing write tid 2 to smithi18128752-16 2024-04-26T01:24:23.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-26T01:24:23.822 INFO:tasks.rados.rados.0.smithi181.stdout:154: finishing copy_from to smithi18128752-50 2024-04-26T01:24:23.822 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 14 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-26T01:24:23.841 INFO:tasks.rados.rados.0.smithi181.stdout:151: finishing write tid 3 to smithi18128752-16 2024-04-26T01:24:23.842 INFO:tasks.rados.rados.0.smithi181.stdout:151: finishing write tid 4 to smithi18128752-16 2024-04-26T01:24:23.851 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:23.852+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:23.924 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 5 to smithi18128752-31 2024-04-26T01:24:23.924 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 31 updating version 0 to 42 2024-04-26T01:24:23.924 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 31 updating version 42 to 43 2024-04-26T01:24:23.924 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 31 version 43 is already newer than 41 2024-04-26T01:24:23.924 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 31 updating version 43 to 44 2024-04-26T01:24:23.924 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 44 (ObjNum 72 snap 9 seq_num 72) dirty exists 2024-04-26T01:24:23.924 INFO:tasks.rados.rados.0.smithi181.stdout:150: left oid 31 (ObjNum 72 snap 9 seq_num 72) 2024-04-26T01:24:23.925 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:23.924+0000 7f22dc60c640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap a after trimming was started 2024-04-26T01:24:23.936 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:394 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.70106997000000004, 2024-04-26T01:24:23.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70109362099999994, 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 394 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240388+0000", 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3410000000000001e-06 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240398+0000", 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8570000000000007e-06 2024-04-26T01:24:23.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240421+0000", 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3198999999999999e-05 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240451+0000", 2024-04-26T01:24:23.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.023e-05 2024-04-26T01:24:23.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727765+0000", 2024-04-26T01:24:23.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.487313618 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727862+0000", 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7324999999999994e-05 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:395 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.70098173200000002, 2024-04-26T01:24:23.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70105936999999996, 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 395 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4199999999999995e-07 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240479+0000", 2024-04-26T01:24:23.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8199999999999996e-06 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240489+0000", 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7089999999999997e-06 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240496+0000", 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9920000000000001e-06 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829732+0000", 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58923568800000004 2024-04-26T01:24:23.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829882+0000", 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015056100000000001 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:396 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.70092539099999995, 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70104310700000005, 2024-04-26T01:24:23.946 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 396 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240531+0000", 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0800000000000004e-07 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240536+0000", 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2059999999999998e-06 2024-04-26T01:24:23.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240547+0000", 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1663e-05 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240555+0000", 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1010000000000003e-06 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831852+0000", 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.59129766500000003 2024-04-26T01:24:23.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832019+0000", 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166773 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 52/44 ECSubReadReply(tid=74, attrs_read=0))", 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.831313+0000", 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.110142355, 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.110295512, 2024-04-26T01:24:23.950 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831313+0000", 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831313+0000", 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831313+0000", 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.15e-07 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831505+0000", 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000191464 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831523+0000", 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7901e-05 2024-04-26T01:24:23.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.931616+0000", 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10009304400000001 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 51/44 ECSubReadReply(tid=73, attrs_read=0))", 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.929100+0000", 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.012355895, 2024-04-26T01:24:23.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.012535457, 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929100+0000", 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929100+0000", 2024-04-26T01:24:23.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929100+0000", 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9199999999999999e-07 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929852+0000", 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00075116400000000002 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929879+0000", 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7061999999999999e-05 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929902+0000", 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2903000000000001e-05 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.956 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 51/44 ECSubReadReply(tid=74, attrs_read=0))", 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.929936+0000", 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.011519305000000001, 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011723535, 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929936+0000", 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929936+0000", 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929937+0000", 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3e-06 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.930132+0000", 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000194902 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.930157+0000", 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4204999999999999e-05 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.930171+0000", 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4966999999999999e-05 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 52/44 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-26T01:24:23.959 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.933167+0000", 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0082887250000000003, 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0085173379999999993, 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.933167+0000", 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.933167+0000", 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.933168+0000", 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3e-06 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.933334+0000", 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166181 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.933362+0000", 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7682e-05 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.933380+0000", 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8142999999999999e-05 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 52/44 ECSubReadReply(tid=75, attrs_read=0))", 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.934108+0000", 2024-04-26T01:24:23.962 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0073473840000000002, 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0075999120000000003, 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.934108+0000", 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.934108+0000", 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.934110+0000", 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5600000000000001e-06 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.934219+0000", 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000109652 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.934250+0000", 2024-04-26T01:24:23.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0332e-05 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 52/44 ECSubWriteReply(tid=77, last_complete=52'46, committed=1, applied=1))", 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.936334+0000", 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0051213910000000003, 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053962869999999996, 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.936334+0000", 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.936334+0000", 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.936335+0000", 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4670000000000001e-06 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.936344+0000", 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8859999999999993e-06 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.936372+0000", 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7739e-05 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.936392+0000", 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0196000000000001e-05 2024-04-26T01:24:23.967 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.968 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:23.968 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.968 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:23.968 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:23.968 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 9 2024-04-26T01:24:23.968 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:23.969 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:24:23.970 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:23.969+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:24.046 INFO:tasks.rados.rados.0.smithi181.stdout:151: finishing write tid 5 to smithi18128752-16 2024-04-26T01:24:24.046 INFO:tasks.rados.rados.0.smithi181.stdout:151: oid 16 updating version 0 to 17 2024-04-26T01:24:24.046 INFO:tasks.rados.rados.0.smithi181.stdout:151: oid 16 updating version 17 to 19 2024-04-26T01:24:24.046 INFO:tasks.rados.rados.0.smithi181.stdout:151: oid 16 version 19 is already newer than 18 2024-04-26T01:24:24.047 INFO:tasks.rados.rados.0.smithi181.stdout:151: oid 16 updating version 19 to 20 2024-04-26T01:24:24.047 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 20 (ObjNum 73 snap 9 seq_num 73) dirty exists 2024-04-26T01:24:24.047 INFO:tasks.rados.rados.0.smithi181.stdout:151: left oid 16 (ObjNum 73 snap 9 seq_num 73) 2024-04-26T01:24:24.070 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:24.071+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:24.137 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:24.137 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:24.137 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:24.137 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:24.137 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:24.152 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:24:24.188 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:24:24.202 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:24.204+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:24.226 INFO:tasks.rados.rados.0.smithi181.stdout:150: done (5 left) 2024-04-26T01:24:24.227 INFO:tasks.rados.rados.0.smithi181.stdout:151: done (4 left) 2024-04-26T01:24:24.227 INFO:tasks.rados.rados.0.smithi181.stdout:152: done (3 left) 2024-04-26T01:24:24.227 INFO:tasks.rados.rados.0.smithi181.stdout:154: done (2 left) 2024-04-26T01:24:24.227 INFO:tasks.rados.rados.0.smithi181.stdout:156: done (1 left) 2024-04-26T01:24:24.227 INFO:tasks.rados.rados.0.smithi181.stdout:157: snap_remove snap 7 2024-04-26T01:24:24.271 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:24.272+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:24.282 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:24.282 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:129 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.711658231000001, 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.91360868200000001, 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.283 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 129 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.284 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575106+0000", 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.043e-06 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575111+0000", 2024-04-26T01:24:24.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3370000000000001e-06 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575128+0000", 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7600000000000001e-05 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575138+0000", 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-04-26T01:24:24.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.877914+0000", 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30277600900000001 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.878073+0000", 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015861299999999999 2024-04-26T01:24:24.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979098+0000", 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101025446 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979375+0000", 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000277309 2024-04-26T01:24:24.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488714+0000", 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50933870599999997 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.289 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:142 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.177122226000002, 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.88158457700000004, 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 142 2024-04-26T01:24:24.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109644+0000", 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0970000000000002e-06 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110339+0000", 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00069456900000000002 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110373+0000", 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.358e-05 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110388+0000", 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5726000000000002e-05 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181183+0000", 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.070795134999999995 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181419+0000", 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023538399999999999 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.182735+0000", 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013160540000000001 2024-04-26T01:24:24.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589098+0000", 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40636337 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991098+0000", 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401999885 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991226+0000", 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012777700000000001 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:143 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:24.296 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.170954562999999, 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87574223200000001, 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 143 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115811+0000", 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.317e-06 2024-04-26T01:24:24.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116338+0000", 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052673300000000004 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116375+0000", 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6922999999999999e-05 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116392+0000", 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6629999999999998e-05 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284095+0000", 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16770365000000001 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284265+0000", 2024-04-26T01:24:24.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016969199999999999 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285568+0000", 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001302554 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589101+0000", 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30353306000000002 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991441+0000", 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40234051300000001 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991551+0000", 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011016 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.302 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:144 3.8s0 3:1fe6984d:::smithi18128752-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.170344168, 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87544766600000001, 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.303 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 144 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116421+0000", 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.15e-06 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116427+0000", 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9070000000000001e-06 2024-04-26T01:24:24.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116443+0000", 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-05 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116452+0000", 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0890000000000006e-06 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286741+0000", 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170288786 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286883+0000", 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142683 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.287610+0000", 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000727172 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589207+0000", 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30159656099999999 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991768+0000", 2024-04-26T01:24:24.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40256125399999998 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991867+0000", 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8814000000000006e-05 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:145 3.8s0 3:1fe6984d:::smithi18128752-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:24.309 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.170298062000001, 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98038499800000001, 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 145 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5600000000000004e-07 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116470+0000", 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116484+0000", 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3837e-05 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.216615+0000", 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100131365 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992549+0000", 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77593446499999996 2024-04-26T01:24:24.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992720+0000", 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017066299999999999 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096851+0000", 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104130502 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.314 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.998333661, 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60172694400000004, 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288438+0000", 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1620000000000008e-06 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288450+0000", 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1922999999999999e-05 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288486+0000", 2024-04-26T01:24:24.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5905999999999998e-05 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388601+0000", 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10011487600000001 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890125+0000", 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50152462099999995 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890157+0000", 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1455999999999998e-05 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.898115762, 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50193089099999999, 2024-04-26T01:24:24.319 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:24.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388649+0000", 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0270000000000001e-06 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388657+0000", 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0260000000000007e-06 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388677+0000", 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0069e-05 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488777+0000", 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100100183 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890549+0000", 2024-04-26T01:24:24.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40177214999999999 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890579+0000", 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9436000000000001e-05 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=15, last_complete=35'9, committed=1, applied=1))", 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.418129+0000", 2024-04-26T01:24:24.323 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.868635012999999, 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47214447599999998, 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418129+0000", 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418129+0000", 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418130+0000", 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.877e-06 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418140+0000", 2024-04-26T01:24:24.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6870000000000006e-06 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418171+0000", 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0479000000000001e-05 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418185+0000", 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890245+0000", 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47205976100000002 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890273+0000", 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7966e-05 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.327 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.868550734999999, 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47216240900000001, 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418213+0000", 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1399999999999997e-07 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418217+0000", 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.676e-06 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418228+0000", 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.062e-05 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418235+0000", 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7130000000000005e-06 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890351+0000", 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47211553299999998 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890375+0000", 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4352999999999999e-05 2024-04-26T01:24:24.331 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.418703+0000", 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.868060127, 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47176985500000002, 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418703+0000", 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418703+0000", 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418705+0000", 2024-04-26T01:24:24.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6440000000000001e-06 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418714+0000", 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2820000000000002e-06 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418744+0000", 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9173e-05 2024-04-26T01:24:24.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.418758+0000", 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4593000000000001e-05 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890448+0000", 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.47169030000000001 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890473+0000", 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4862999999999999e-05 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.797863197000002, 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50191135200000003, 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.336 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488902+0000", 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9369999999999998e-06 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488913+0000", 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0789e-05 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488954+0000", 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0416000000000001e-05 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589090+0000", 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100136138 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990779+0000", 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401689352 2024-04-26T01:24:24.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990812+0000", 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2719999999999998e-05 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:160 3.8s0 3:1945891d:::smithi18128752-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.782148061000001, 2024-04-26T01:24:24.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.79222307800000002, 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 160 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504618+0000", 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5600000000000001e-06 2024-04-26T01:24:24.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504629+0000", 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0956e-05 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504661+0000", 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2428999999999997e-05 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504678+0000", 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6422999999999998e-05 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093363+0000", 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58868559200000004 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093514+0000", 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015052300000000001 2024-04-26T01:24:24.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.094369+0000", 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00085476900000000001 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096327+0000", 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001957781 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:24.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296743+0000", 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20041606200000001 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296839+0000", 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5983000000000006e-05 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.346 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:269 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.113041897, 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81822368899999998, 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 269 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.347 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173723+0000", 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4399999999999999e-07 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173738+0000", 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5798999999999999e-05 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173754+0000", 2024-04-26T01:24:24.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5943e-05 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173763+0000", 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1730000000000001e-06 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385572+0000", 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.211808092 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385724+0000", 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015250999999999999 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589244+0000", 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203520116 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589408+0000", 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016355699999999999 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.991945+0000", 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40253765499999999 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:383 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~720896] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.0660163540000001, 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50440791100000004, 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 383 2024-04-26T01:24:24.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220749+0000", 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.995e-06 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221593+0000", 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00084381899999999999 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221614+0000", 2024-04-26T01:24:24.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1061999999999999e-05 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221623+0000", 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3130000000000004e-06 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.317149+0000", 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095525231000000002 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.317292+0000", 2024-04-26T01:24:24.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014299199999999999 2024-04-26T01:24:24.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318544+0000", 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001252567 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322759+0000", 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042146029999999999 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:24.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.724936+0000", 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40217761699999999 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725155+0000", 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000218712 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.359 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:384 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~49152] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.0651201589999999, 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50390550300000003, 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 384 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221644+0000", 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0200000000000001e-07 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221675+0000", 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0929999999999997e-05 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221683+0000", 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3559999999999993e-06 2024-04-26T01:24:24.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221689+0000", 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7370000000000003e-06 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.319416+0000", 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.097727075999999996 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.319496+0000", 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9579000000000005e-05 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.320065+0000", 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00056965599999999998 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322760+0000", 2024-04-26T01:24:24.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026947189999999999 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725421+0000", 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40266085000000001 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725549+0000", 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012789799999999999 2024-04-26T01:24:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:385 3.8s0 3:14c0c421:::smithi18128752-31:head [setxattr _header (56) in=7b] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.065064638, 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50445409900000004, 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.366 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 385 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.82e-07 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221702+0000", 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.779e-06 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221715+0000", 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2337000000000001e-05 2024-04-26T01:24:24.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221721+0000", 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.207e-06 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.321880+0000", 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100158994 2024-04-26T01:24:24.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.321983+0000", 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103669 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322488+0000", 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050421999999999999 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.423268+0000", 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10077992299999999 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725960+0000", 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30269248500000001 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.726153+0000", 2024-04-26T01:24:24.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000193003 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:386 3.8s0 3:14c0c421:::smithi18128752-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.06503466, 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70800976199999999, 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.373 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 386 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221732+0000", 2024-04-26T01:24:24.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1129999999999999e-06 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221738+0000", 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4609999999999996e-06 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221744+0000", 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3539999999999996e-06 2024-04-26T01:24:24.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727164+0000", 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50541931600000001 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727250+0000", 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6618000000000003e-05 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929739+0000", 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20248843399999999 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:394 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.0463781539999999, 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.79330826399999999, 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.378 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 394 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:24.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240388+0000", 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3410000000000001e-06 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240398+0000", 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8570000000000007e-06 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240421+0000", 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3198999999999999e-05 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240451+0000", 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.023e-05 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727765+0000", 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.487313618 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727862+0000", 2024-04-26T01:24:24.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7324999999999994e-05 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.033694+0000", 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30583169399999999 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:395 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:24.383 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.0462899160000001, 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.89472404000000005, 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 395 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.384 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4199999999999995e-07 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240479+0000", 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8199999999999996e-06 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240489+0000", 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7089999999999997e-06 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240496+0000", 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9920000000000001e-06 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829732+0000", 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58923568800000004 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829882+0000", 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015056100000000001 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.135198+0000", 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.305315528 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:396 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.046233575, 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99608940400000001, 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 396 2024-04-26T01:24:24.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240531+0000", 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0800000000000004e-07 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240536+0000", 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2059999999999998e-06 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240547+0000", 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1663e-05 2024-04-26T01:24:24.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240555+0000", 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1010000000000003e-06 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831852+0000", 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.59129766500000003 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:24.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832019+0000", 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166773 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.236619+0000", 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40460038799999998 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.393 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:24.395 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:24:24.396 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:24.396+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:24.496 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:24.497+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:24.550 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:24.571 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:24.571 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:24.571 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-04-26T01:24:24.571 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:24.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:24.596 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:24.601+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:24.698 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:24.699+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:24.705 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:24.705 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:24.713 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:24.713 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:24.713 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:24.714 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:24.714 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:24.727 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:24.727 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:24.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:24.748 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:24.748 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:24.766 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.5 to 1 (10000) 2024-04-26T01:24:24.767 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-04-26T01:24:24.772 INFO:tasks.rados.rados.0.smithi181.stdout:157: done (1 left) 2024-04-26T01:24:24.773 INFO:tasks.rados.rados.0.smithi181.stdout:158: snap_create 2024-04-26T01:24:24.773 INFO:tasks.rados.rados.0.smithi181.stdout:153: expect (ObjNum 47 snap 0 seq_num 47) 2024-04-26T01:24:24.775 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:24.776 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:24.792 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-26T01:24:24.794 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:24.794 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:24.798 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:24.803+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:24.816 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:24.816 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:24.839 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:24.840 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:24.863 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:24.863 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:24.864 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:24.864 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:24.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:24.864 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:24.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.864 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/30 ECSubWrite(tid=39, reqid=client.4589.0:136, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.782240+0000", 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 19.084621009999999, 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10402797799999999, 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782240+0000", 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782240+0000", 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782242+0000", 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3530000000000002e-06 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782690+0000", 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00044834200000000001 2024-04-26T01:24:24.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782722+0000", 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2169999999999999e-05 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.782739+0000", 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6546e-05 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.883637+0000", 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100898232 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.883660+0000", 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3243000000000001e-05 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.886192+0000", 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002531971 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.886268+0000", 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5121000000000001e-05 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/30 ECSubWrite(tid=40, reqid=client.4589.0:137, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.783289+0000", 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 19.083571805999998, 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103977202, 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.869 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.783289+0000", 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.783289+0000", 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.783290+0000", 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.66e-06 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.783330+0000", 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9462000000000003e-05 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.783363+0000", 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3096000000000003e-05 2024-04-26T01:24:24.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.783377+0000", 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4436e-05 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.885275+0000", 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10189770500000001 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.885290+0000", 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4936000000000001e-05 2024-04-26T01:24:24.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.887169+0000", 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001878563 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.887266+0000", 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7343999999999995e-05 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 35/34 ECSubWrite(tid=15, reqid=client.4589.0:85, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.378885+0000", 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.487975333000001, 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104150651, 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.378885+0000", 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.378885+0000", 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.378887+0000", 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.339e-06 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379143+0000", 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00025659700000000001 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379166+0000", 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3102999999999999e-05 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379185+0000", 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8359000000000002e-05 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.479455+0000", 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100270659 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.479470+0000", 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4949e-05 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482945+0000", 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0034743899999999999 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483036+0000", 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1255000000000006e-05 2024-04-26T01:24:24.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 35/34 ECSubWrite(tid=16, reqid=client.4589.0:86, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.379223+0000", 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.487637616000001, 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10385730999999999, 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.878 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379223+0000", 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379223+0000", 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379224+0000", 2024-04-26T01:24:24.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2399999999999996e-07 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379268+0000", 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3769000000000001e-05 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379293+0000", 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5389e-05 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379318+0000", 2024-04-26T01:24:24.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.442e-05 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.480573+0000", 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101255 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.480589+0000", 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6843000000000001e-05 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483045+0000", 2024-04-26T01:24:24.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024560189999999998 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483080+0000", 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4946000000000001e-05 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 35/34 ECSubWrite(tid=17, reqid=client.4589.0:146, at_version=35'9, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.379351+0000", 2024-04-26T01:24:24.882 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.487509199000002, 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103757008, 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379351+0000", 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379351+0000", 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379352+0000", 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4399999999999999e-07 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379587+0000", 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023498400000000001 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379619+0000", 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1681999999999999e-05 2024-04-26T01:24:24.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379660+0000", 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1523000000000002e-05 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.480917+0000", 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101256297 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.480930+0000", 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2935999999999999e-05 2024-04-26T01:24:24.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483083+0000", 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021530569999999999 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483108+0000", 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5684999999999999e-05 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.379704+0000", 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.487156690999999, 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104018789, 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379704+0000", 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379704+0000", 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379705+0000", 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8700000000000004e-07 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379925+0000", 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000219748 2024-04-26T01:24:24.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379946+0000", 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1769000000000001e-05 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.379959+0000", 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2313000000000001e-05 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482013+0000", 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10205395 2024-04-26T01:24:24.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482028+0000", 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5270000000000001e-05 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483655+0000", 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016270060000000001 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483723+0000", 2024-04-26T01:24:24.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7846000000000001e-05 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.es1 35/34 ECSubRead(tid=23, to_read={3:734f3d85:::smithi18128752-49:head=0,4096,0}, subchunks={3:734f3d85:::smithi18128752-49:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.311836+0000", 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.555024913, 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20030075899999999, 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.311836+0000", 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.311836+0000", 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.311837+0000", 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:24:24.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.311847+0000", 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0555e-05 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.311864+0000", 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6748999999999998e-05 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.311874+0000", 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0165e-05 2024-04-26T01:24:24.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.511899+0000", 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200024851 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.512136+0000", 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023747199999999999 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s6 36/34 ECSubWrite(tid=14, reqid=client.4589.0:171, at_version=36'8, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.338479+0000", 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.528381851999999, 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17369040699999999, 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338479+0000", 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338479+0000", 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338479+0000", 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4400000000000001e-07 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338509+0000", 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9836e-05 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338541+0000", 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1844999999999997e-05 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338566+0000", 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4839999999999999e-05 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338756+0000", 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018998100000000001 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.338764+0000", 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2640000000000008e-06 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.512141+0000", 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17337749599999999 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.512169+0000", 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7801e-05 2024-04-26T01:24:24.898 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s6 37/34 ECSubWrite(tid=16, reqid=client.4589.0:181, at_version=37'9, trim_to=0'0, roll_forward_to=36'8))", 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.794974670999999, 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20273409100000001, 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.072132+0000", 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024541999999999999 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.072149+0000", 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7192000000000001e-05 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172254+0000", 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100104756 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172426+0000", 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017276600000000001 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172443+0000", 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7136e-05 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.274572+0000", 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102128393 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.274620+0000", 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8152999999999998e-05 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.7s2 37/30 ECSubRead(tid=54, to_read={3:e3d33b00:::smithi18128752-5:head=0,20480,0}, subchunks={3:e3d33b00:::smithi18128752-5:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.079572+0000", 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.787288411999999, 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17339227099999999, 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.079572+0000", 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.079572+0000", 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.079573+0000", 2024-04-26T01:24:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2800000000000002e-07 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.079580+0000", 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4490000000000002e-06 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.079592+0000", 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1885999999999999e-05 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.079599+0000", 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1969999999999999e-06 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.252750+0000", 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.17315085499999999 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.252964+0000", 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021445599999999999 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s6 37/34 ECSubWrite(tid=17, reqid=client.4589.0:182, at_version=37'10, trim_to=0'0, roll_forward_to=36'8))", 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.172314+0000", 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.694546635999998, 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10235172300000001, 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172314+0000", 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172314+0000", 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172315+0000", 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2669999999999999e-06 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172543+0000", 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000227321 2024-04-26T01:24:24.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172576+0000", 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3772000000000001e-05 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172595+0000", 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8187000000000001e-05 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.273530+0000", 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100935871 2024-04-26T01:24:24.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.273539+0000", 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0080000000000008e-06 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.274634+0000", 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010946619999999999 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.274666+0000", 2024-04-26T01:24:24.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1634999999999998e-05 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.7s2 41/30 ECSubWrite(tid=96, reqid=osd.4.0:96, at_version=41'20, trim_to=0'0, roll_forward_to=39'18))", 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.505001+0000", 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.361859483, 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103028521, 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505001+0000", 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505001+0000", 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505003+0000", 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8059999999999999e-06 2024-04-26T01:24:24.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505098+0000", 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4957999999999994e-05 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505129+0000", 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1062999999999997e-05 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505153+0000", 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3649999999999999e-05 2024-04-26T01:24:24.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505393+0000", 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023990799999999999 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.505417+0000", 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4632000000000001e-05 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607939+0000", 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102522266 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.608030+0000", 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0237999999999999e-05 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 41/34 ECSubWrite(tid=46, reqid=osd.2.0:46, at_version=41'29, trim_to=0'0, roll_forward_to=41'27))", 2024-04-26T01:24:24.915 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.599843+0000", 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.267018083, 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20203437599999999, 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599843+0000", 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599843+0000", 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599844+0000", 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6840000000000001e-06 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599881+0000", 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6418999999999998e-05 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599908+0000", 2024-04-26T01:24:24.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7053e-05 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599925+0000", 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7278999999999998e-05 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800207+0000", 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200281828 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800221+0000", 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4039999999999999e-05 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.801788+0000", 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015674650000000001 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.801877+0000", 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8608000000000003e-05 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.919 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s6 41/34 ECSubWrite(tid=28, reqid=osd.0.0:27, at_version=0'0, trim_to=0'0, roll_forward_to=41'17))", 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.607423+0000", 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.259438092, 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.193444807, 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607423+0000", 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607423+0000", 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607423+0000", 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1999999999999998e-07 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607517+0000", 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3524999999999996e-05 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607556+0000", 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8809000000000001e-05 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.607580+0000", 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3770000000000001e-05 2024-04-26T01:24:24.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.708575+0000", 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100995643 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.708590+0000", 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4673e-05 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800780+0000", 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.092190253 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800867+0000", 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7213999999999995e-05 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 45/44 ECSubWrite(tid=41, reqid=client.4589.0:294, at_version=45'25, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:24.924 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:17.239863+0000", 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 7.6269980860000004, 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104615043, 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.239863+0000", 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.239863+0000", 2024-04-26T01:24:24.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.239864+0000", 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6190000000000001e-06 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.240145+0000", 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028073099999999999 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.240164+0000", 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9315999999999999e-05 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.240175+0000", 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0556e-05 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.340581+0000", 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100406385 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.340614+0000", 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2845999999999997e-05 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.344410+0000", 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0037964890000000001 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:17.344478+0000", 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7100999999999993e-05 2024-04-26T01:24:24.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/30 ECSubWrite(tid=125, reqid=client.4589.0:326, at_version=47'29, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.559058+0000", 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.3078021509999997, 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103022455, 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559058+0000", 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559058+0000", 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559060+0000", 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0040000000000002e-06 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559135+0000", 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4542000000000006e-05 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559178+0000", 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3381999999999999e-05 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559203+0000", 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4879e-05 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.659439+0000", 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10023573299999999 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.659466+0000", 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7157000000000001e-05 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.662042+0000", 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0025761949999999999 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.662081+0000", 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8563000000000003e-05 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.7s2 47/30 ECSubWrite(tid=126, reqid=client.4589.0:327, at_version=47'30, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.559258+0000", 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.3076027510000001, 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10286003000000001, 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.933 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559258+0000", 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559258+0000", 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559259+0000", 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.093e-06 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559295+0000", 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6208999999999998e-05 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559320+0000", 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5092999999999998e-05 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.659475+0000", 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10015450100000001 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.660145+0000", 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000670033 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.660174+0000", 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9272999999999999e-05 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.662087+0000", 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001912509 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.662118+0000", 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1319000000000003e-05 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 47/44 ECSubWrite(tid=54, reqid=osd.5.0:54, at_version=47'29, trim_to=0'0, roll_forward_to=47'27))", 2024-04-26T01:24:24.937 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:19.180755+0000", 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.6861058060000005, 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102653959, 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.180755+0000", 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.180755+0000", 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.180757+0000", 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9870000000000002e-06 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.180811+0000", 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4051999999999997e-05 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.180852+0000", 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1279999999999998e-05 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.180876+0000", 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4136000000000001e-05 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.281121+0000", 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100244687 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.281149+0000", 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.792e-05 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.283176+0000", 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020269279999999999 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:19.283409+0000", 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000232969 2024-04-26T01:24:24.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s3 51/44 ECSubRead(tid=73, to_read={3:1fe6984d:::smithi18128752-48:7=0,200704,0}, subchunks={3:1fe6984d:::smithi18128752-48:7=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.728275+0000", 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.138585819, 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19916468200000001, 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.728275+0000", 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.728275+0000", 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.728276+0000", 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4959999999999999e-06 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.728294+0000", 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7520000000000002e-05 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.728333+0000", 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8865000000000001e-05 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.828420+0000", 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100087254 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.926855+0000", 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098434782999999998 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.927439+0000", 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00058476399999999999 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s3 52/44 ECSubRead(tid=75, to_read={3:1fe6984d:::smithi18128752-48:7=0,200704,0}, subchunks={3:1fe6984d:::smithi18128752-48:7=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.832379+0000", 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.03448118, 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19859623500000001, 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832379+0000", 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832379+0000", 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832381+0000", 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.167e-06 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832393+0000", 2024-04-26T01:24:24.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2196e-05 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832421+0000", 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7846000000000001e-05 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832440+0000", 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9797e-05 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:24.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.030739+0000", 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19829897899999999 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.030976+0000", 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023625 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:24.949 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:24.950 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:24.951 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:24.951 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:24.952 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:24.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:24.953 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:24.951+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:25.001 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:25.002 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:25.023 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:25.024 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:25.046 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:25.046 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:25.052 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:25.055+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:25.072 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:25.073 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:25.089 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:25.089 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:25.100 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:25.100 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:25.104 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:25.126 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:25.126 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:25.154 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:25.153+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:25.232 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:25.233 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:25.247 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:25.254 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:25.255+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:25.355 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:25.355+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:25.373 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:25.373 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:25.389 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:25.456 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:25.459+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:25.522 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:25.522 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:25.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:25.557 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:25.559+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:25.654 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:25.654 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:25.657 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:25.661+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:25.669 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:25.758 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:25.757+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:25.776 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T01:24:25.783 INFO:tasks.rados.rados.0.smithi181.stdout:153: done (1 left) 2024-04-26T01:24:25.784 INFO:tasks.rados.rados.0.smithi181.stdout:158: done (0 left) 2024-04-26T01:24:25.784 INFO:tasks.rados.rados.0.smithi181.stdout:159: read oid 46 snap -1 2024-04-26T01:24:25.784 INFO:tasks.rados.rados.0.smithi181.stdout:159: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:25.784 INFO:tasks.rados.rados.0.smithi181.stdout:160: read oid 7 snap 2 2024-04-26T01:24:25.784 INFO:tasks.rados.rados.0.smithi181.stdout:160: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-26T01:24:25.784 INFO:tasks.rados.rados.0.smithi181.stdout:161: snap_create 2024-04-26T01:24:25.788 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:25.788 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:25.802 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:25.803 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-26T01:24:25.859 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:25.861+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:25.924 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:25.924 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:25.939 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:25.960 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:25.963+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:26.061 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:26.063+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:26.062 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:26.062 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:26.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:26.094 INFO:tasks.rados.rados.0.smithi181.stdout:159: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:26.162 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:26.163+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:26.200 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:26.200 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:26.214 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:26.263 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:26.267+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:26.290 INFO:tasks.rados.rados.0.smithi181.stdout:160: expect (ObjNum 6 snap 0 seq_num 6) 2024-04-26T01:24:26.339 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:26.339 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:26.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:26.396 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:26.399+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:26.464 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:26.467+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:26.469 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:26.469 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:26.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:26.565 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:26.567+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:26.640 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:26.640 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:26.654 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:26.700 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:26.703+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:26.767 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:26.769+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:26.768 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:26.768 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:26.781 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:26.786 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.8 to 1 (10000) 2024-04-26T01:24:26.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:24:26.795 INFO:tasks.rados.rados.0.smithi181.stdout:159: done (2 left) 2024-04-26T01:24:26.795 INFO:tasks.rados.rados.0.smithi181.stdout:160: done (1 left) 2024-04-26T01:24:26.796 INFO:tasks.rados.rados.0.smithi181.stdout:161: done (0 left) 2024-04-26T01:24:26.796 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 13 current snap is 11 2024-04-26T01:24:26.796 INFO:tasks.rados.rados.0.smithi181.stdout:162: seq_num 75 ranges {212992=491520,704512=770048} 2024-04-26T01:24:26.807 INFO:tasks.rados.rados.0.smithi181.stdout:162: writing smithi18128752-13 from 212992 to 704512 tid 1 2024-04-26T01:24:26.814 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-26T01:24:26.821 INFO:tasks.rados.rados.0.smithi181.stdout:162: writing smithi18128752-13 from 704512 to 1474560 tid 2 2024-04-26T01:24:26.821 INFO:tasks.rados.rados.0.smithi181.stdout:163: copy_from oid 15 from oid 5 current snap is 11 2024-04-26T01:24:26.821 INFO:tasks.rados.rados.0.smithi181.stdout:164: rollback oid 19 current snap is 11 2024-04-26T01:24:26.821 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 19 to 10 2024-04-26T01:24:26.822 INFO:tasks.rados.rados.0.smithi181.stdout:165: copy_from oid 21 from oid 24 current snap is 11 2024-04-26T01:24:26.822 INFO:tasks.rados.rados.0.smithi181.stdout:166: copy_from oid 3 from oid 7 current snap is 11 2024-04-26T01:24:26.822 INFO:tasks.rados.rados.0.smithi181.stdout:167: read oid 33 snap -1 2024-04-26T01:24:26.822 INFO:tasks.rados.rados.0.smithi181.stdout:167: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:26.822 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 16 current snap is 11 2024-04-26T01:24:26.822 INFO:tasks.rados.rados.0.smithi181.stdout:168: seq_num 76 ranges {1490944=507904,1998848=507904,2506752=262144} 2024-04-26T01:24:26.852 INFO:tasks.rados.rados.0.smithi181.stdout:168: writing smithi18128752-16 from 1490944 to 1998848 tid 1 2024-04-26T01:24:26.858 INFO:tasks.rados.rados.0.smithi181.stdout:168: writing smithi18128752-16 from 1998848 to 2506752 tid 2 2024-04-26T01:24:26.862 INFO:tasks.rados.rados.0.smithi181.stdout:168: writing smithi18128752-16 from 2506752 to 2768896 tid 3 2024-04-26T01:24:26.862 INFO:tasks.rados.rados.0.smithi181.stdout:169: rmattr oid 34 current snap is 11 2024-04-26T01:24:26.862 INFO:tasks.rados.rados.0.smithi181.stdout:169: done (7 left) 2024-04-26T01:24:26.862 INFO:tasks.rados.rados.0.smithi181.stdout:170: snap_remove snap 9 2024-04-26T01:24:26.907 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:26.911+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:26.942 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:26.942 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:26.956 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:26.968 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:26.968+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:27.068 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:27.068+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:27.072 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:27.073 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:27.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:27.147 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:27.170 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:27.172+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:27.223 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:27.223 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:27.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:27.270 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:27.272+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:27.299 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:27.299 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:27.320 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:27.321 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:27.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:27.342 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:27.363 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:27.363 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:27.363 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:27.363 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:27.363 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:27.363 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:27.364 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:27.371 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:27.375+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:27.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:27.381 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:27.381 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:27.402 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:27.402 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:27.422 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:27.422 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:27.442 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:27.442 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:27.461 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:27.461 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:27.472 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:27.477+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:27.482 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:27.504 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:27.504 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s6 35/34 ECSubWrite(tid=165, reqid=client.4589.0:113, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:05.384986+0000", 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.122053860000001, 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10190149900000001, 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.505 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.384986+0000", 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.384986+0000", 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.384988+0000", 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7e-06 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.385019+0000", 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1606000000000002e-05 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.385045+0000", 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6356e-05 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.385068+0000", 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3005000000000001e-05 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.485605+0000", 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100536415 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.485621+0000", 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5962999999999999e-05 2024-04-26T01:24:27.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.486809+0000", 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011885660000000001 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.486887+0000", 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7887999999999997e-05 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.509 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 35/30 ECSubWrite(tid=37, reqid=client.4589.0:121, at_version=35'5, trim_to=0'0, roll_forward_to=34'4))", 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:05.596405+0000", 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.910634594000001, 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102485587, 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596405+0000", 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596405+0000", 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596407+0000", 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.804e-06 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596661+0000", 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025394000000000003 2024-04-26T01:24:27.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.596684+0000", 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3278999999999999e-05 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.696819+0000", 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100134475 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.696987+0000", 2024-04-26T01:24:27.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016887299999999999 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.697009+0000", 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1081999999999999e-05 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.698821+0000", 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018124619999999999 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:05.698891+0000", 2024-04-26T01:24:27.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9672000000000006e-05 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 35/30 ECSubWrite(tid=42, reqid=client.4589.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.086757+0000", 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.420282374999999, 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101553475, 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086757+0000", 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086757+0000", 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086759+0000", 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7129999999999999e-06 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086779+0000", 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0403000000000001e-05 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.086809+0000", 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9085999999999999e-05 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186953+0000", 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100144019 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.187126+0000", 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000173105 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.187154+0000", 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8218999999999999e-05 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.188230+0000", 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010764279999999999 2024-04-26T01:24:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.188311+0000", 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0501999999999996e-05 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:169 3.as0 3:56e8b2fd:::smithi18128752-12:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.209064304999998, 2024-04-26T01:24:27.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.415445014, 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 169 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297980+0000", 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3359999999999997e-06 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297998+0000", 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8499999999999999e-05 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298041+0000", 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3171999999999999e-05 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398202+0000", 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10016098800000001 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398693+0000", 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049036800000000003 2024-04-26T01:24:27.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398913+0000", 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220051 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.610194+0000", 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.211280998 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612042+0000", 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018484860000000001 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713242+0000", 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10119953 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713420+0000", 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017858500000000001 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.523 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:170 3.as0 3:56e8b2fd:::smithi18128752-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.108799522999998, 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.213653854, 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 170 2024-04-26T01:24:27.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398242+0000", 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0530000000000001e-06 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398252+0000", 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4830000000000005e-06 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398271+0000", 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-05 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398282+0000", 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1476e-05 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404421+0000", 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006138592 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404504+0000", 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2750999999999994e-05 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611404+0000", 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20690029200000001 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611712+0000", 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030829799999999998 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611894+0000", 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018167899999999999 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.528 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:212 3.as0 3:56e8b2fd:::smithi18128752-12:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.386040873999999, 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.306234695, 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212 2024-04-26T01:24:27.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121002+0000", 2024-04-26T01:24:27.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.366e-06 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121071+0000", 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9202000000000006e-05 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121106+0000", 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5052999999999998e-05 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121125+0000", 2024-04-26T01:24:27.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8219e-05 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221415+0000", 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029012800000001 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221427+0000", 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.187e-05 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222099+0000", 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067277899999999998 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222322+0000", 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000222272 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.224059+0000", 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017374059999999999 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226116+0000", 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020565679999999999 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426988+0000", 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20087223500000001 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.427234+0000", 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024559700000000001 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 37/30 ECSubWriteReply(tid=6, last_complete=37'5, committed=1, applied=1))", 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.226125+0000", 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.280914202000002, 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.158071511, 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226125+0000", 2024-04-26T01:24:27.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226125+0000", 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226128+0000", 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4870000000000001e-06 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226140+0000", 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1566000000000001e-05 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226171+0000", 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1316000000000003e-05 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326267+0000", 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009647000000001 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384168+0000", 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.057900419000000002 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384197+0000", 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9252999999999999e-05 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 37/30 ECSubWriteReply(tid=6, last_complete=37'5, committed=1, applied=1))", 2024-04-26T01:24:27.538 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.326457+0000", 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.180582234999999, 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101657281, 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326457+0000", 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326457+0000", 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326459+0000", 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6670000000000001e-06 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326467+0000", 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9160000000000001e-06 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.326485+0000", 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8050000000000002e-05 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426618+0000", 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100133342 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426762+0000", 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014365299999999999 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.428115+0000", 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013526530000000001 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.281285911000001, 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28191508700000001, 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.542 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225756+0000", 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.756e-06 2024-04-26T01:24:27.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225765+0000", 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225784+0000", 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.325876+0000", 2024-04-26T01:24:27.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100092051 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507611+0000", 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.181735758 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507669+0000", 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7361999999999997e-05 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.199983164999999, 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044763199999999, 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307065+0000", 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9039999999999992e-06 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307072+0000", 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5619999999999998e-06 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307078+0000", 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1830000000000002e-06 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507467+0000", 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200389598 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507504+0000", 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6879999999999999e-05 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.181002464999999, 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.182176437, 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.549 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326037+0000", 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326038+0000", 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0920000000000001e-06 2024-04-26T01:24:27.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326046+0000", 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2339999999999997e-06 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326060+0000", 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4187e-05 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.326083+0000", 2024-04-26T01:24:27.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3640000000000001e-05 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507750+0000", 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.18166618600000001 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.508214+0000", 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046409799999999997 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.178054701000001, 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19916183800000001, 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328985+0000", 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328986+0000", 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0839999999999999e-06 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.328992+0000", 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8130000000000001e-06 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.329007+0000", 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4769e-05 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.329014+0000", 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4240000000000004e-06 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.528083+0000", 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19906906299999999 2024-04-26T01:24:27.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.528147+0000", 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3684999999999998e-05 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.104223362000001, 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20097827700000001, 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.556 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402817+0000", 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.026e-06 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402832+0000", 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4684999999999999e-05 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402851+0000", 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9338e-05 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402863+0000", 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1832e-05 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.503005+0000", 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014197399999999 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.503027+0000", 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1903999999999999e-05 2024-04-26T01:24:27.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.603709+0000", 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10068225 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.603795+0000", 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5267999999999997e-05 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.560 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 41/30 ECSubWrite(tid=94, reqid=osd.4.0:94, at_version=41'18, trim_to=0'0, roll_forward_to=39'14))", 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.002977332, 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202154895, 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504063+0000", 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4900000000000001e-07 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504085+0000", 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2342999999999999e-05 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504093+0000", 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504100+0000", 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2470000000000002e-06 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704414+0000", 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20031389399999999 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704439+0000", 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.499e-05 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706155+0000", 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017162480000000001 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706217+0000", 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1878999999999995e-05 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.564 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(1.0 46/45 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 34'157, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@34'157, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:17.801826+0000", 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.7052139939999993, 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102200655, 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.801826+0000", 2024-04-26T01:24:27.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.801826+0000", 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.801827+0000", 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.44e-06 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.802351+0000", 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052414599999999997 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.802393+0000", 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1596e-05 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.802425+0000", 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1695000000000001e-05 2024-04-26T01:24:27.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.902775+0000", 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100350232 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.902797+0000", 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2521999999999999e-05 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:17.904026+0000", 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001229024 2024-04-26T01:24:27.568 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.bs5 47/34 ECSubWrite(tid=122, reqid=client.4589.0:331, at_version=47'8, trim_to=0'0, roll_forward_to=41'6))", 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.370345+0000", 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.1366947029999999, 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10321674, 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.370345+0000", 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.370345+0000", 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.370346+0000", 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4e-07 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.370724+0000", 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000378549 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.370755+0000", 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0318999999999999e-05 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.470861+0000", 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100105951 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.471057+0000", 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019675199999999999 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.471077+0000", 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9959999999999999e-05 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.473474+0000", 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002396614 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.473562+0000", 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7655e-05 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/30 ECSubWrite(tid=97, reqid=client.4589.0:400, at_version=52'20, trim_to=0'0, roll_forward_to=51'18))", 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.575108084, 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199335239, 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.573 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:27.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8000000000000001e-07 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931970+0000", 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8256000000000001e-05 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931998+0000", 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8186000000000001e-05 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.932022+0000", 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3269999999999999e-05 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.129435+0000", 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19741354999999999 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.129451+0000", 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6195999999999999e-05 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.131180+0000", 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017288760000000001 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.131267+0000", 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6525000000000003e-05 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s2 52/44 ECSubWrite(tid=212, reqid=osd.13.0:212, at_version=52'22, trim_to=0'0, roll_forward_to=51'20))", 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.153679+0000", 2024-04-26T01:24:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.35336072, 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10235354300000001, 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.153679+0000", 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.153679+0000", 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.153681+0000", 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3130000000000001e-06 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.153722+0000", 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0493e-05 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.153753+0000", 2024-04-26T01:24:27.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1551999999999999e-05 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.153778+0000", 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.429e-05 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.254022+0000", 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10024472500000001 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.254044+0000", 2024-04-26T01:24:27.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1189999999999999e-05 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.255944+0000", 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001900718 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.256033+0000", 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8262000000000003e-05 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/30 ECSubWrite(tid=98, reqid=client.4589.0:400, at_version=0'0, trim_to=0'0, roll_forward_to=52'20))", 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.232749+0000", 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.2742906720000002, 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101696843, 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.232749+0000", 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.232749+0000", 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.232751+0000", 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1459999999999999e-06 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.232776+0000", 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4513999999999999e-05 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.232809+0000", 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3510000000000003e-05 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.232832+0000", 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2989e-05 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.333112+0000", 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100279759 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.333128+0000", 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5656000000000001e-05 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.334344+0000", 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012159440000000001 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.334446+0000", 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010232499999999999 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.585 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s2 52/44 ECSubWrite(tid=213, reqid=osd.13.0:212, at_version=0'0, trim_to=0'0, roll_forward_to=52'22))", 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.1493690590000001, 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30091122399999998, 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:27.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357672+0000", 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.75e-06 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:27.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357708+0000", 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5781999999999997e-05 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357774+0000", 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6131999999999996e-05 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.457877+0000", 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100102935 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:27.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458051+0000", 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017357000000000001 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458066+0000", 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5526e-05 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658511+0000", 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044448300000001 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:27.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658582+0000", 2024-04-26T01:24:27.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1045999999999997e-05 2024-04-26T01:24:27.590 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.590 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.590 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.590 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:27.590 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:27.590 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:27.591 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:27.591 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:27.592 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:27.592 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:27.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:27.593 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:27.593 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:27.594 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:27.594 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:27.594 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:27.591+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:27.609 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:27.609 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:27.692 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:27.695+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:27.762 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:27.762 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:27.762 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:27.762 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:27.776 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:27.789 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:24:27.789 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.9 to 1 (10000) 2024-04-26T01:24:27.794 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:27.795+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:27.795 INFO:tasks.rados.rados.0.smithi181.stdout:170: done (7 left) 2024-04-26T01:24:27.795 INFO:tasks.rados.rados.0.smithi181.stdout:171: read oid 39 snap -1 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:171: expect (ObjNum 53 snap 1 seq_num 53) 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:172: rmattr oid 4 current snap is 11 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:172: done (8 left) 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:173: snap_create 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:168: finishing write tid 1 to smithi18128752-16 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:165: finishing copy_from racing read to smithi18128752-21 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:168: finishing write tid 2 to smithi18128752-16 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:168: finishing write tid 3 to smithi18128752-16 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:168: finishing write tid 4 to smithi18128752-16 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:162: finishing write tid 1 to smithi18128752-13 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:162: finishing write tid 2 to smithi18128752-13 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:162: finishing write tid 3 to smithi18128752-13 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:164: finishing rollback tid 1 to smithi18128752-19 2024-04-26T01:24:27.796 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 25 (ObjNum 33 snap 0 seq_num 1953066355) dirty dne 2024-04-26T01:24:27.797 INFO:tasks.rados.rados.0.smithi181.stdout:167: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:27.815 INFO:tasks.rados.rados.0.smithi181.stdout:166: finishing copy_from racing read to smithi18128752-3 2024-04-26T01:24:27.815 INFO:tasks.rados.rados.0.smithi181.stdout:165: finishing copy_from to smithi18128752-21 2024-04-26T01:24:27.815 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 21 v 28 (ObjNum 71 snap 7 seq_num 71) dirty exists 2024-04-26T01:24:27.815 INFO:tasks.rados.rados.0.smithi181.stdout:163: finishing copy_from racing read to smithi18128752-15 2024-04-26T01:24:27.816 INFO:tasks.rados.rados.0.smithi181.stdout:162: finishing write tid 4 to smithi18128752-13 2024-04-26T01:24:27.816 INFO:tasks.rados.rados.0.smithi181.stdout:162: oid 13 updating version 0 to 33 2024-04-26T01:24:27.816 INFO:tasks.rados.rados.0.smithi181.stdout:162: oid 13 version 33 is already newer than 32 2024-04-26T01:24:27.816 INFO:tasks.rados.rados.0.smithi181.stdout:162: oid 13 version 33 is already newer than 31 2024-04-26T01:24:27.816 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 33 (ObjNum 75 snap 11 seq_num 75) dirty exists 2024-04-26T01:24:27.816 INFO:tasks.rados.rados.0.smithi181.stdout:162: left oid 13 (ObjNum 75 snap 11 seq_num 75) 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:166: finishing copy_from to smithi18128752-3 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 11 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:168: finishing write tid 5 to smithi18128752-16 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:168: oid 16 updating version 0 to 26 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:168: oid 16 updating version 26 to 29 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:168: oid 16 version 29 is already newer than 28 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:168: oid 16 version 29 is already newer than 27 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 29 (ObjNum 76 snap 11 seq_num 76) dirty exists 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:168: left oid 16 (ObjNum 76 snap 11 seq_num 76) 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:163: finishing copy_from to smithi18128752-15 2024-04-26T01:24:27.817 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-04-26T01:24:27.818 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-26T01:24:27.895 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:27.895+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:27.898 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:27.898 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:27.899 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:27.899 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:27.899 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:27.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:24:27.995 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:27.995+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:28.076 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:28.077 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:28.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:28.077 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2024-04-26T01:24:28.077 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:28.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:24:28.096 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:28.099+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:28.197 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:28.199+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:28.210 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:28.210 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:28.210 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:28.211 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:28.225 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:24:28.298 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:28.299+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:28.308 INFO:tasks.rados.rados.0.smithi181.stdout:171: expect (ObjNum 53 snap 1 seq_num 53) 2024-04-26T01:24:28.351 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:28.351 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:28.351 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:28.351 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:28.352 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:28.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:24:28.399 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:28.399+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:28.492 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:28.492 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:28.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:28.492 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:28.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.492 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:150 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:28.492 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.316941342, 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3959368480000001, 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 150 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179064+0000", 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7269999999999999e-06 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179608+0000", 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00054408799999999997 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179641+0000", 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2873000000000002e-05 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179658+0000", 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6863000000000001e-05 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268330+0000", 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0886726819999999 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268562+0000", 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023123100000000001 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.270316+0000", 2024-04-26T01:24:28.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017541189999999999 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474438+0000", 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20412262 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574871+0000", 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100432788 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574998+0000", 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126857 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:151 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.310566479999999, 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.494027684, 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:28.498 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 151 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:28.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185438+0000", 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2170000000000001e-06 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185895+0000", 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000456291 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185927+0000", 2024-04-26T01:24:28.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1989000000000002e-05 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185942+0000", 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5075999999999999e-05 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272106+0000", 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0861648319999999 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272243+0000", 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013630399999999999 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.273909+0000", 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016665930000000001 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474512+0000", 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20060244399999999 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679353+0000", 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20484124000000001 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679464+0000", 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110698 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:152 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:28.503 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.310034573999999, 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4937803999999999, 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 152 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185969+0000", 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.167e-06 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185975+0000", 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5029999999999997e-06 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185986+0000", 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1283e-05 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185994+0000", 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6060000000000003e-06 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372230+0000", 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.186235766 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372376+0000", 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014574 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.373451+0000", 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010752890000000001 2024-04-26T01:24:28.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474514+0000", 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101062945 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679632+0000", 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20511794799999999 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679748+0000", 2024-04-26T01:24:28.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116153 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:153 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.309996198, 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.900776316, 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.509 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 153 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186007+0000", 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6700000000000002e-07 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186011+0000", 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5559999999999999e-06 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186025+0000", 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3873000000000001e-05 2024-04-26T01:24:28.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186035+0000", 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0526000000000001e-05 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475079+0000", 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2890438200000001 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475255+0000", 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000176279 2024-04-26T01:24:28.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680464+0000", 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20520917499999999 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680595+0000", 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013023099999999999 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086783+0000", 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40618798900000003 2024-04-26T01:24:28.513 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.037339333999999, 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.407182076, 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458664+0000", 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458680+0000", 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5953000000000001e-05 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458693+0000", 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2972999999999999e-05 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458701+0000", 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4829999999999999e-06 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.865517+0000", 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40681550500000002 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.865845+0000", 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032856200000000002 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:164 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:28.517 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.198089736, 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78562828299999998, 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:28.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297916+0000", 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297932+0000", 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2542e-05 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297993+0000", 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7813e-05 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475329+0000", 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17733589199999999 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475446+0000", 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117074 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083541+0000", 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60809543499999996 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:165 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.197974322, 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78769006399999997, 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.522 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298030+0000", 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.59e-06 2024-04-26T01:24:28.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298039+0000", 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1470000000000007e-06 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298064+0000", 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5542e-05 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298080+0000", 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5279999999999999e-05 2024-04-26T01:24:28.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476021+0000", 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.177941503 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476177+0000", 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155873 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085718+0000", 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60954112900000001 2024-04-26T01:24:28.525 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:166 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.197898281000001, 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78831037500000001, 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.526 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 166 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298106+0000", 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-06 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298113+0000", 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7570000000000004e-06 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298128+0000", 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5e-05 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298139+0000", 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0986e-05 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476693+0000", 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17855336099999999 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476835+0000", 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014191599999999999 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086415+0000", 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60957998499999999 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/34 ECSubReadReply(tid=23, attrs_read=0))", 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.895380254999999, 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.48295446400000003, 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.530 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600624+0000", 2024-04-26T01:24:28.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6199999999999999e-06 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601039+0000", 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041512500000000002 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601068+0000", 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.906e-05 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701180+0000", 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10011215699999999 2024-04-26T01:24:28.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.082862+0000", 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.381681457 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083577+0000", 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071504499999999998 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:189 3.cs0 3:395c8678:::smithi18128752-25:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.418962668999999, 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61527076999999997, 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.534 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077044+0000", 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0570000000000003e-06 2024-04-26T01:24:28.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077061+0000", 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6616000000000002e-05 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077097+0000", 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6829000000000003e-05 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077120+0000", 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2779999999999999e-05 2024-04-26T01:24:28.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086851+0000", 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097310599999999997 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.087007+0000", 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015570199999999999 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.495435+0000", 2024-04-26T01:24:28.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40842789299999999 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497946+0000", 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025112519999999998 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692090+0000", 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19414421300000001 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692311+0000", 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220368 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:190 3.cs0 3:395c8678:::smithi18128752-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.418846532, 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.42078353899999998, 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.539 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077158+0000", 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7540000000000001e-06 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077167+0000", 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2459999999999992e-06 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077188+0000", 2024-04-26T01:24:28.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1403000000000002e-05 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077357+0000", 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000168985 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187444+0000", 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110086218 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187592+0000", 2024-04-26T01:24:28.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014808599999999999 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497585+0000", 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.309993713 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497788+0000", 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020263099999999999 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497940+0000", 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015150299999999999 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:199 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.418602831000001, 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41377173299999997, 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.544 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:28.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.956e-06 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077413+0000", 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1056999999999999e-05 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077435+0000", 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2602000000000001e-05 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7252999999999999e-05 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187721+0000", 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110268303 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187913+0000", 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000191845 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.491171+0000", 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30325871700000001 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:200 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:28.548 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.418525847000002, 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41453494499999999, 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 200 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.364e-06 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077486+0000", 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8159999999999994e-06 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077505+0000", 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9360000000000001e-05 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077521+0000", 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5685999999999999e-05 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188598+0000", 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111076802 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188834+0000", 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023592099999999999 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.492012+0000", 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30317799600000001 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.552 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:201 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.418462555000001, 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41550727300000001, 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 201 2024-04-26T01:24:28.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077541+0000", 2024-04-26T01:24:28.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.314e-06 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077548+0000", 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8390000000000001e-06 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077566+0000", 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7893e-05 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077581+0000", 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5290000000000001e-05 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.189776+0000", 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112194944 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.190248+0000", 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047181099999999999 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.493047+0000", 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.302799182 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:321 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.149726+0000", 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.346276851000001, 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.405632877, 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.557 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 321 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149726+0000", 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149726+0000", 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149729+0000", 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9000000000000002e-06 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149741+0000", 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2862e-05 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149773+0000", 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1983999999999997e-05 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149791+0000", 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7422e-05 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.234698+0000", 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084906827000000004 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:28.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.234724+0000", 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6466e-05 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.350693+0000", 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.115968881 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.350993+0000", 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000299654 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.555359+0000", 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.204365881 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:322 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.3461815, 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.707643833, 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.562 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 322 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149822+0000", 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.004e-06 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149829+0000", 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.613e-06 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149840+0000", 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1819e-05 2024-04-26T01:24:28.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249944+0000", 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100103704 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250099+0000", 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015437400000000001 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250119+0000", 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0392999999999999e-05 2024-04-26T01:24:28.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352060+0000", 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101941083 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352219+0000", 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015886700000000001 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857465+0000", 2024-04-26T01:24:28.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50524597599999999 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:323 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.246019690000001, 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.70925194000000003, 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.567 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 323 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249984+0000", 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4559999999999999e-06 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249994+0000", 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0186e-05 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250012+0000", 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7836999999999999e-05 2024-04-26T01:24:28.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250023+0000", 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1143e-05 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254128+0000", 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041046069999999997 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254145+0000", 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7405999999999999e-05 2024-04-26T01:24:28.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452144+0000", 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.197998339 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452432+0000", 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028853800000000002 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.959235+0000", 2024-04-26T01:24:28.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50680242799999997 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s6 47/30 ECSubWrite(tid=123, reqid=client.4589.0:325, at_version=47'28, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.039235173, 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50042386699999997, 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456768+0000", 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7599999999999997e-07 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456800+0000", 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2240000000000003e-05 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456819+0000", 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8799000000000001e-05 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456835+0000", 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5860000000000001e-05 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.655702+0000", 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19886722300000001 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.655714+0000", 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.218e-05 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:28.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957130+0000", 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301416354 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957191+0000", 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0735e-05 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s6 47/30 ECSubWrite(tid=125, reqid=client.4589.0:326, at_version=47'29, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:24:28.576 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.034997409000001, 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.496226999, 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461007+0000", 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7689999999999999e-06 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461040+0000", 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3563999999999999e-05 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461069+0000", 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8989e-05 2024-04-26T01:24:28.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.561206+0000", 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10013631100000001 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857800+0000", 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29659451599999997 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857828+0000", 2024-04-26T01:24:28.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7583e-05 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957201+0000", 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099372708000000004 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957232+0000", 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1559000000000001e-05 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:397 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [copy-from ver 14] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2552039199999996, 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.58707355400000005, 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 397 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:28.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240802+0000", 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5200000000000002e-06 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240819+0000", 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6444999999999999e-05 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240855+0000", 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6702999999999997e-05 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240873+0000", 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7900000000000001e-05 2024-04-26T01:24:28.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.241008+0000", 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013451000000000001 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.241021+0000", 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3504e-05 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.245054+0000", 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004032926 2024-04-26T01:24:28.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.245220+0000", 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016579 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.553412+0000", 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30819260700000001 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.556948+0000", 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035356580000000001 2024-04-26T01:24:28.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.827661+0000", 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.270712428 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.827872+0000", 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021156300000000001 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:28.586 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:28.587 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:28.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:24:28.589 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:28.591+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:28.689 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:28.691+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:28.716 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:28.716 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:28.716 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:28.716 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:28.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:24:28.767 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.10 to 1 (10000) 2024-04-26T01:24:28.772 INFO:tasks.rados.rados.0.smithi181.stdout:162: done (8 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:163: done (7 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:164: done (6 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:165: done (5 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:166: done (4 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:167: done (3 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:168: done (2 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:171: done (1 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:173: done (0 left) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:174: copy_from oid 17 from oid 8 current snap is 12 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:175: read oid 32 snap -1 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:175: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-26T01:24:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:176: snap_remove snap 11 2024-04-26T01:24:28.790 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:28.791+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:28.793 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-26T01:24:28.853 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:28.853 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:28.853 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:28.853 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:28.853 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:28.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:24:28.912 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:28.917+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:28.992 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:28.991+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:29.034 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:29.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 35/30 ECSubWrite(tid=15, reqid=client.4589.0:142, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.183393+0000", 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.854117596999998, 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102129083, 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183393+0000", 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183393+0000", 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183396+0000", 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4669999999999998e-06 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183618+0000", 2024-04-26T01:24:29.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022251700000000001 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183651+0000", 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2456000000000002e-05 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.183678+0000", 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7742999999999998e-05 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.283894+0000", 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10021598 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.283913+0000", 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8895000000000001e-05 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.285439+0000", 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015254000000000001 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.285522+0000", 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3625e-05 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.039 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/34 ECSubWrite(tid=11, reqid=client.4589.0:158, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.735791+0000", 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.301719976000001, 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101625586, 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735791+0000", 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735791+0000", 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735792+0000", 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.553e-06 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735830+0000", 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7635999999999997e-05 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735860+0000", 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9512000000000001e-05 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.735875+0000", 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5353999999999999e-05 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.836096+0000", 2024-04-26T01:24:29.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100221269 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.836121+0000", 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4655999999999999e-05 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.837346+0000", 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001225304 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.837416+0000", 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0302000000000006e-05 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s4 36/30 ECSubWrite(tid=172, reqid=client.4589.0:167, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:29.047 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.629880773, 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102386068, 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407630+0000", 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.03e-07 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407939+0000", 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030856200000000002 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407967+0000", 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7869e-05 2024-04-26T01:24:29.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.407983+0000", 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6404000000000001e-05 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.508232+0000", 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100248772 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.508247+0000", 2024-04-26T01:24:29.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4663e-05 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.509938+0000", 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001691084 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.510016+0000", 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8311000000000003e-05 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/34 ECSubWrite(tid=19, reqid=client.4589.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=37'11))", 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.756317110000001, 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103193194, 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:29.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281196+0000", 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.046e-06 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281226+0000", 2024-04-26T01:24:29.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9762999999999999e-05 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281269+0000", 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3059000000000002e-05 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281295+0000", 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6432999999999999e-05 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281433+0000", 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013830900000000001 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281449+0000", 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5846999999999999e-05 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384326+0000", 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102877183 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384387+0000", 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0554000000000003e-05 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.6s3 37/34 ECSubRead(tid=20, to_read={3:69f31b93:::smithi18128752-39:head=0,4096,0}, subchunks={3:69f31b93:::smithi18128752-39:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.282222+0000", 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.755289126000001, 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10090128600000001, 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.056 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282222+0000", 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282222+0000", 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282224+0000", 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0200000000000001e-06 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282246+0000", 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1936000000000001e-05 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282285+0000", 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9431999999999998e-05 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.282317+0000", 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1622999999999999e-05 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.382647+0000", 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100330423 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.383123+0000", 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047585200000000002 2024-04-26T01:24:29.059 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 41/30 ECSubWrite(tid=35, reqid=client.4589.0:268, at_version=41'21, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.283921+0000", 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.753589820999998, 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101082875, 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283921+0000", 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283921+0000", 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283924+0000", 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6539999999999999e-06 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.283969+0000", 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5082e-05 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.284005+0000", 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5836e-05 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.284024+0000", 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9760000000000001e-05 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.384220+0000", 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100195435 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.384228+0000", 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5080000000000005e-06 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.384974+0000", 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000745716 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.385004+0000", 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9884000000000001e-05 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 41/34 ECSubWrite(tid=28, reqid=osd.0.0:27, at_version=0'0, trim_to=0'0, roll_forward_to=41'17))", 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.412602+0000", 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.624909182, 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10092269800000001, 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.064 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412602+0000", 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412602+0000", 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412603+0000", 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.392e-06 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412616+0000", 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2778e-05 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.412631+0000", 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4938e-05 2024-04-26T01:24:29.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.512740+0000", 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100108817 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.512822+0000", 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2826999999999998e-05 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.512836+0000", 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.364e-05 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.513487+0000", 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065126699999999997 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.513524+0000", 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7039000000000002e-05 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.541516303000002, 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10316732200000001, 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495996+0000", 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.457e-06 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496024+0000", 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8269999999999999e-05 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496047+0000", 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3045e-05 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496077+0000", 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0046999999999999e-05 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.596334+0000", 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10025700899999999 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.596353+0000", 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8423e-05 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.599074+0000", 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002720943 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.599162+0000", 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8127999999999994e-05 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/34 ECSubWrite(tid=29, reqid=osd.0.0:29, at_version=43'19, trim_to=0'0, roll_forward_to=41'17))", 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:14.111706+0000", 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.925804475, 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10200825099999999, 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.073 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.111706+0000", 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.111706+0000", 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.111707+0000", 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1600000000000001e-07 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.111764+0000", 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6755999999999997e-05 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.111797+0000", 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2848999999999997e-05 2024-04-26T01:24:29.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.211898+0000", 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010137099999999 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.212038+0000", 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013969300000000001 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.212063+0000", 2024-04-26T01:24:29.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5303000000000002e-05 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.213604+0000", 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015411120000000001 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.213715+0000", 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110451 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.077 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/34 ECSubWrite(tid=30, reqid=osd.0.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=43'19))", 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:14.315851+0000", 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.721659503, 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103015009, 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315851+0000", 2024-04-26T01:24:29.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315851+0000", 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315853+0000", 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9970000000000001e-06 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315886+0000", 2024-04-26T01:24:29.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2526e-05 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315928+0000", 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2502000000000001e-05 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.416043+0000", 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100114561 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.416195+0000", 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151793 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.416223+0000", 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7838999999999999e-05 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.418768+0000", 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025458199999999999 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.418866+0000", 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7971000000000001e-05 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 47/30 ECSubWrite(tid=74, reqid=client.4589.0:332, at_version=47'15, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.46234269, 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20225293799999999, 2024-04-26T01:24:29.082 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575170+0000", 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.08e-06 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575220+0000", 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0246e-05 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575252+0000", 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1482000000000001e-05 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.675393+0000", 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014118599999999 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775690+0000", 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029639899999999 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775706+0000", 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6133e-05 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777345+0000", 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016393899999999999 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777421+0000", 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6021999999999998e-05 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=56, reqid=client.4589.0:336, at_version=48'32, trim_to=0'0, roll_forward_to=47'29))", 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.292829+0000", 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.7446823130000002, 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10234890200000001, 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.292829+0000", 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.292829+0000", 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.292830+0000", 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.697e-06 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.292936+0000", 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000106164 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.292966+0000", 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9646999999999999e-05 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.393055+0000", 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10008861400000001 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.393262+0000", 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020761900000000001 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.393278+0000", 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5818999999999999e-05 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.395105+0000", 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001826523 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.395177+0000", 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2818999999999995e-05 2024-04-26T01:24:29.090 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 48/44 ECSubWrite(tid=83, reqid=client.4589.0:343, at_version=48'23, trim_to=0'0, roll_forward_to=46'20))", 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.472738+0000", 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.564772434, 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101255768, 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.472738+0000", 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.472738+0000", 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.472739+0000", 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0200000000000001e-07 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.472765+0000", 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5972000000000001e-05 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.472778+0000", 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2993000000000001e-05 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.472790+0000", 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1892000000000001e-05 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.572948+0000", 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10015745299999999 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.572962+0000", 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4905999999999999e-05 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.573951+0000", 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00098846000000000003 2024-04-26T01:24:29.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.573994+0000", 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3290000000000001e-05 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=58, reqid=client.4589.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.4378497320000001, 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301426421, 2024-04-26T01:24:29.095 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599662+0000", 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.299e-06 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599675+0000", 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2605e-05 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599692+0000", 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7221999999999999e-05 2024-04-26T01:24:29.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699796+0000", 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100104162 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699955+0000", 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000158319 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699983+0000", 2024-04-26T01:24:29.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8453e-05 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901051+0000", 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201067725 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901088+0000", 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6636e-05 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=60, reqid=osd.5.0:60, at_version=48'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.3375663160000002, 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20186222600000001, 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9200000000000002e-07 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699973+0000", 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8328999999999999e-05 2024-04-26T01:24:29.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699980+0000", 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0770000000000002e-06 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699990+0000", 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.011e-05 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900620+0000", 2024-04-26T01:24:29.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20062928899999999 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900633+0000", 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2994e-05 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901770+0000", 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001137482 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901807+0000", 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6653000000000003e-05 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 51/44 ECSubWrite(tid=67, reqid=client.4589.0:382, at_version=51'41, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.8216252959999997, 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103342327, 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3700000000000001e-07 2024-04-26T01:24:29.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.216245+0000", 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00035873300000000002 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.216275+0000", 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0528999999999999e-05 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.216293+0000", 2024-04-26T01:24:29.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7827000000000001e-05 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.316559+0000", 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100266483 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.316576+0000", 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6328999999999999e-05 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.319160+0000", 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025844570000000001 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.319228+0000", 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7632000000000006e-05 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 53/30 ECSubWrite(tid=99, reqid=osd.14.0:99, at_version=53'22, trim_to=0'0, roll_forward_to=52'20))", 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.887406+0000", 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1501043700000002, 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10178619999999999, 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.887406+0000", 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.887406+0000", 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.887414+0000", 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.379e-06 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.887453+0000", 2024-04-26T01:24:29.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9226e-05 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.887482+0000", 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8526000000000001e-05 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987617+0000", 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100135602 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987754+0000", 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013638999999999999 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987770+0000", 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6273000000000002e-05 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.989116+0000", 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013466249999999999 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.989193+0000", 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6179000000000004e-05 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 53/44 ECSubWrite(tid=79, reqid=osd.5.0:79, at_version=53'48, trim_to=0'0, roll_forward_to=52'46))", 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.987815+0000", 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0496962510000003, 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10179057800000001, 2024-04-26T01:24:29.113 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987815+0000", 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987815+0000", 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987815+0000", 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6300000000000004e-07 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987871+0000", 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5226000000000003e-05 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987901+0000", 2024-04-26T01:24:29.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0616e-05 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.987924+0000", 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2926e-05 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:25.088121+0000", 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100196961 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:25.088141+0000", 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9697000000000001e-05 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:25.089510+0000", 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0013695580000000001 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:25.089605+0000", 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4831000000000001e-05 2024-04-26T01:24:29.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.6s3 56/34 ECSubRead(tid=39, to_read={3:69f31b93:::smithi18128752-39:head=0,225280,0}, subchunks={3:69f31b93:::smithi18128752-39:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:27.907541+0000", 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.1299701980000001, 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101988723, 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.118 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:27.907541+0000", 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:27.907541+0000", 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:27.907542+0000", 2024-04-26T01:24:29.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.403e-06 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:27.907557+0000", 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5432999999999999e-05 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:27.907589+0000", 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1062999999999997e-05 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.007696+0000", 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100107212 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.009394+0000", 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016986869999999999 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.009529+0000", 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013492500000000001 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 56/34 ECSubWrite(tid=42, reqid=osd.0.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=56'27))", 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:28.517146+0000", 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.520365262, 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101146397, 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.517146+0000", 2024-04-26T01:24:29.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.517146+0000", 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.517148+0000", 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.526e-06 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:29.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.517179+0000", 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0669000000000001e-05 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.517212+0000", 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3469000000000001e-05 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.617330+0000", 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100117783 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.617460+0000", 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129493 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.617469+0000", 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1279999999999997e-06 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.618235+0000", 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00076660499999999996 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:29.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:28.618292+0000", 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6724000000000001e-05 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:29.126 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:29.128 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:24:29.129 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:29.127+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:29.228 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:29.231+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:29.256 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:29.257 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:29.257 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:29.257 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:29.272 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:24:29.329 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:29.333+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:29.400 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:29.400 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:29.401 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:29.401 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:29.401 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:29.414 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:24:29.429 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:29.431+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:29.530 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:29.531+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:29.543 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:29.543 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:29.543 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:29.543 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:29.543 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.543 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:115 3.6s0 3:69f31b93:::smithi18128752-39:head [append 0~638976,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.543 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 24.159715205000001, 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.27460739099999998, 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 115 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:29.544 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385705+0000", 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5990000000000001e-06 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.545 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386759+0000", 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010545940000000001 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386792+0000", 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2376999999999998e-05 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386816+0000", 2024-04-26T01:24:29.546 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4680000000000001e-05 2024-04-26T01:24:29.547 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.547 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.547 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.547 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.454087+0000", 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.067270658999999997 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.454285+0000", 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019791700000000001 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.455558+0000", 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001272731 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.548 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558702+0000", 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103144082 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.659941+0000", 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101238869 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.660311+0000", 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000369883 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.549 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:116 3.6s0 3:69f31b93:::smithi18128752-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 24.158563459, 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.37357116200000001, 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 116 2024-04-26T01:24:29.550 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.551 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386857+0000", 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.703e-06 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386866+0000", 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5670000000000001e-06 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386887+0000", 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1316000000000001e-05 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.552 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.554 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386904+0000", 2024-04-26T01:24:29.554 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6767000000000001e-05 2024-04-26T01:24:29.554 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.556964+0000", 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17005975700000001 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.557176+0000", 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021186600000000001 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.555 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558110+0000", 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00093395700000000002 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.559561+0000", 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014508430000000001 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.759894+0000", 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033284600000001 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.556 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.760426+0000", 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00053253999999999997 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:117 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 24.158488806000001, 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77605625899999997, 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:29.557 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 117 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:29.558 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386931+0000", 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3599999999999999e-06 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386939+0000", 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8369999999999997e-06 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.559 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386957+0000", 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8683000000000001e-05 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386974+0000", 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558747+0000", 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17177335099999999 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.560 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558933+0000", 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018625099999999999 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761103+0000", 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202169231 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761231+0000", 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000128734 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.561 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162986+0000", 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.401754422 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 35/30 ECSubWrite(tid=12, reqid=client.4589.0:128, at_version=35'8, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.580804+0000", 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 23.964614412, 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20196134800000001, 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.562 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580804+0000", 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580804+0000", 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.563 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580805+0000", 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.494e-06 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580833+0000", 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7373e-05 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.580858+0000", 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4819000000000001e-05 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.564 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.680966+0000", 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100108062 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.781188+0000", 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100222797 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.781217+0000", 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8563000000000001e-05 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.565 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.782727+0000", 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001509727 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.782765+0000", 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8513000000000002e-05 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 35/30 ECSubWrite(tid=15, reqid=client.4589.0:142, at_version=35'9, trim_to=0'0, roll_forward_to=35'8))", 2024-04-26T01:24:29.566 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.216520+0000", 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 23.328898727999999, 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20095770800000001, 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216520+0000", 2024-04-26T01:24:29.567 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216520+0000", 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216521+0000", 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6750000000000001e-06 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.568 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216661+0000", 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014010699999999999 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.216682+0000", 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0182000000000001e-05 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.316787+0000", 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10010555 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.569 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.316920+0000", 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013260799999999999 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.316933+0000", 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3621999999999999e-05 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.417433+0000", 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10049991899999999 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.570 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.417477+0000", 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4045000000000003e-05 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:156 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~475136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 23.047783063000001, 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33900037700000002, 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 156 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.572 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497638+0000", 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2110000000000001e-06 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498467+0000", 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082923399999999998 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.573 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498486+0000", 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9341e-05 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498496+0000", 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8179999999999999e-06 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498610+0000", 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000114259 2024-04-26T01:24:29.574 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498770+0000", 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016017599999999999 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726262+0000", 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22749161900000001 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726429+0000", 2024-04-26T01:24:29.575 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000166825 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.728061+0000", 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016325490000000001 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735773+0000", 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0077114189999999997 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836407+0000", 2024-04-26T01:24:29.576 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10063380399999999 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836636+0000", 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022912200000000001 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:157 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 23.041562596999999, 2024-04-26T01:24:29.577 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.43381848000000001, 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 157 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.578 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503857+0000", 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2920000000000001e-06 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.579 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504200+0000", 2024-04-26T01:24:29.580 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034286599999999999 2024-04-26T01:24:29.580 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.580 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.580 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.580 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504218+0000", 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7961e-05 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504227+0000", 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6240000000000008e-06 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.582 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504381+0000", 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015489900000000001 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504555+0000", 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017342900000000001 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.729729+0000", 2024-04-26T01:24:29.583 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225174034 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.730038+0000", 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030876200000000002 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.732244+0000", 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022065190000000001 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.584 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735776+0000", 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035319330000000001 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937311+0000", 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201534612 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937674+0000", 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00036354900000000001 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.585 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:158 3.6s0 3:60f0095a:::smithi18128752-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 23.041171524999999, 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.630196439, 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.586 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 158 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.587 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504255+0000", 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0989999999999999e-06 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504262+0000", 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5660000000000003e-06 2024-04-26T01:24:29.588 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504266+0000", 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0980000000000004e-06 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504650+0000", 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00038378400000000002 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:29.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504796+0000", 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014592199999999999 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733685+0000", 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22888902699999999 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733975+0000", 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029027399999999998 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.590 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.734990+0000", 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010148500000000001 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836708+0000", 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10171771 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134326+0000", 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29761846199999997 2024-04-26T01:24:29.591 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134443+0000", 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117142 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:159 3.6s0 3:60f0095a:::smithi18128752-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 23.0411459, 2024-04-26T01:24:29.592 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.73307043299999997, 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 159 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.593 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504273+0000", 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3500000000000002e-07 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504274+0000", 2024-04-26T01:24:29.594 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.562e-06 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504281+0000", 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3790000000000003e-06 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504286+0000", 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1989999999999997e-06 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504861+0000", 2024-04-26T01:24:29.595 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00057502400000000004 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504981+0000", 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012016899999999999 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837238+0000", 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33225718900000001 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:29.596 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837352+0000", 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113583 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134907+0000", 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29755462599999999 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.135101+0000", 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019402500000000001 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.597 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.237343+0000", 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102242342 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=11, last_complete=35'7, committed=1, applied=1))", 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.711271834000001, 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30069599499999999, 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.598 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.599 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0800000000000003e-07 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834153+0000", 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1699999999999996e-06 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834168+0000", 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5017999999999999e-05 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.600 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.934244+0000", 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100075936 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134251+0000", 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200007516 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134843+0000", 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059144700000000004 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.601 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:171 3.6s0 3:69f31b93:::smithi18128752-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.208722963, 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36489432300000002, 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 171 2024-04-26T01:24:29.602 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336697+0000", 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3829999999999999e-06 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336701+0000", 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.053e-06 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336710+0000", 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8349999999999993e-06 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.604 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336715+0000", 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5219999999999998e-06 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337072+0000", 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00035695799999999999 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337245+0000", 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017283899999999999 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.605 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.338075+0000", 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082985399999999999 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.339323+0000", 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012477390000000001 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701394+0000", 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36207125499999998 2024-04-26T01:24:29.606 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701590+0000", 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000195885 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s3 41/30 ECSubWrite(tid=97, reqid=osd.4.0:97, at_version=41'22, trim_to=0'0, roll_forward_to=39'18))", 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.038918323000001, 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201986786, 2024-04-26T01:24:29.607 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.608 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506501+0000", 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0100000000000001e-06 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506544+0000", 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2605999999999998e-05 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506567+0000", 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3269999999999999e-05 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.609 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.606669+0000", 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10010197899999999 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.706960+0000", 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10029147400000001 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.706979+0000", 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8403e-05 2024-04-26T01:24:29.610 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.708374+0000", 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013951510000000001 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.708487+0000", 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000112893 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.611 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 51/44 ECSubWrite(tid=71, reqid=client.4589.0:385, at_version=0'0, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.8190095120000001, 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30170434200000001, 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.612 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726411+0000", 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.88e-06 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726432+0000", 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.137e-05 2024-04-26T01:24:29.613 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726464+0000", 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1664e-05 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826555+0000", 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100091667 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826667+0000", 2024-04-26T01:24:29.614 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011122500000000001 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826683+0000", 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6149000000000001e-05 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028057+0000", 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20137464699999999 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.615 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028113+0000", 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5739999999999998e-05 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:430 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801177+0000", 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.7442411330000001, 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.21020058, 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:29.616 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 430 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801177+0000", 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801177+0000", 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801181+0000", 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3840000000000001e-06 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801197+0000", 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6008e-05 2024-04-26T01:24:29.618 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801227+0000", 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0701000000000003e-05 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801242+0000", 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4384e-05 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801378+0000", 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013629700000000001 2024-04-26T01:24:29.619 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801391+0000", 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2628e-05 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.904897+0000", 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103506743 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.620 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.905064+0000", 2024-04-26T01:24:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016662900000000001 2024-04-26T01:24:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.011378+0000", 2024-04-26T01:24:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.106313806 2024-04-26T01:24:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:431 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.744143134, 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40905669700000002, 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.622 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 431 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.623 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801276+0000", 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1399999999999995e-07 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801283+0000", 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4840000000000001e-06 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801299+0000", 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6707000000000001e-05 2024-04-26T01:24:29.624 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801309+0000", 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0112000000000001e-05 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801486+0000", 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017675199999999999 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801494+0000", 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6490000000000008e-06 2024-04-26T01:24:29.625 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906250+0000", 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10475599300000001 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906413+0000", 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016362200000000001 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.210332+0000", 2024-04-26T01:24:29.626 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303918464 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:432 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.7440352059999999, 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51044518999999999, 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.627 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 432 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.628 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801384+0000", 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801390+0000", 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2750000000000001e-06 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801402+0000", 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1747e-05 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.629 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.901507+0000", 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100104586 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.007942+0000", 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10643573100000001 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.008127+0000", 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000185007 2024-04-26T01:24:29.630 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.311828+0000", 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303700894 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.5359640919999999, 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30115782000000002, 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.631 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.632 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009456+0000", 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7799999999999999e-06 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010014+0000", 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00055842399999999996 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010048+0000", 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3407e-05 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.633 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010060+0000", 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.254e-05 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310534+0000", 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30047343599999998 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310612+0000", 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8232999999999994e-05 2024-04-26T01:24:29.634 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.53302103, 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.298319318, 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.635 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.636 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012425+0000", 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7229999999999998e-05 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012432+0000", 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9160000000000001e-06 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.637 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310652+0000", 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29821998799999999 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310717+0000", 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4183999999999994e-05 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.638 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.5167556879999999, 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.28320577800000002, 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.639 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028664+0000", 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.266e-06 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028857+0000", 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019285299999999999 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.640 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028876+0000", 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9584000000000001e-05 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.128964+0000", 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100087615 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310808+0000", 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18184441400000001 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.641 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.311868+0000", 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001060046 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 56/44 ECSubWrite(tid=84, reqid=osd.5.0:83, at_version=0'0, trim_to=0'0, roll_forward_to=56'52))", 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.047176+0000", 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.4982425930000001, 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201033192, 2024-04-26T01:24:29.642 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.047176+0000", 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.047176+0000", 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.643 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.047177+0000", 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4470000000000001e-06 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.047196+0000", 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9113000000000001e-05 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.047224+0000", 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7610000000000002e-05 2024-04-26T01:24:29.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.047239+0000", 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4766999999999999e-05 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.147476+0000", 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10023710800000001 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.645 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.147491+0000", 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5667000000000001e-05 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.248126+0000", 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10063472699999999 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.248209+0000", 2024-04-26T01:24:29.646 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2752999999999999e-05 2024-04-26T01:24:29.647 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.647 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.647 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.647 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.647 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.647 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:29.647 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:29.649 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:24:29.650 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:29.647+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:29.750 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:29.751+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:29.769 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.11 to 1 (10000) 2024-04-26T01:24:29.774 INFO:tasks.rados.rados.0.smithi181.stdout:176: done (2 left) 2024-04-26T01:24:29.774 INFO:tasks.rados.rados.0.smithi181.stdout:177: rmattr oid 13 current snap is 12 2024-04-26T01:24:29.775 INFO:tasks.rados.rados.0.smithi181.stdout:177: done (2 left) 2024-04-26T01:24:29.775 INFO:tasks.rados.rados.0.smithi181.stdout:178: copy_from oid 27 from oid 29 current snap is 12 2024-04-26T01:24:29.775 INFO:tasks.rados.rados.0.smithi181.stdout:175: expect (ObjNum 31 snap 0 seq_num 31) 2024-04-26T01:24:29.794 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-26T01:24:29.813 INFO:tasks.rados.rados.0.smithi181.stdout:174: finishing copy_from racing read to smithi18128752-17 2024-04-26T01:24:29.814 INFO:tasks.rados.rados.0.smithi181.stdout:175: done (2 left) 2024-04-26T01:24:29.814 INFO:tasks.rados.rados.0.smithi181.stdout:179: snap_remove snap 10 2024-04-26T01:24:29.820 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:434 3.7s0 3:e9e07e85:::smithi18128752-17:head [copy-from ver 29 in=54b] snapc d=[d,c,8,4] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.0594330110000001, 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.059465922, 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:29.833 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 434 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:29.834 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778655+0000", 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8770000000000002e-06 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778667+0000", 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1888999999999999e-05 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.835 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778697+0000", 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0205999999999998e-05 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778714+0000", 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6467999999999999e-05 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878928+0000", 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100213993 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878940+0000", 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2724999999999999e-05 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882181+0000", 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032411419999999998 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882351+0000", 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169371 2024-04-26T01:24:29.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.678907+0000", 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.79655591199999998 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 57/30 ECSubWriteReply(tid=168, last_complete=57'38, committed=1, applied=1))", 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:29.783625+0000", 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.054460635, 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.054594480000000001, 2024-04-26T01:24:29.838 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.783625+0000", 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.783625+0000", 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.783625+0000", 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2299999999999999e-07 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.783632+0000", 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5520000000000001e-06 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.783647+0000", 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5424e-05 2024-04-26T01:24:29.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.783656+0000", 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5029999999999997e-06 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 2 2024-04-26T01:24:29.841 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:29.850 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:24:29.851 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:29.852 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:29.852 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:29.855+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:29.873 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:29.874 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:29.912 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:29.912 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:29.933 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:29.933 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:29.952 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:29.951+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:29.974 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:29.974 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:29.974 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:29.974 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:29.993 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:24:30.038 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:30.039 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:30.054 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:30.057+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:30.064 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:30.065 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:30.100 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:30.100 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:30.136 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:30.136 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:30.155 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:30.155 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:30.155 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:30.159+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:30.169 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:24:30.170 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:30.170 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:30.206 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:30.206 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:30.237 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:30.237 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:30.256 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:30.259+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:30.258 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:30.258 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:30.274 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:30.275 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:30.279 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:30.280 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:30.293 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:24:30.294 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:30.295 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:30.315 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:30.315 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:30.336 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:30.336 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:30.357 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:30.359+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:30.408 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:30.408 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:30.422 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:24:30.458 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:30.463+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:30.543 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:30.544 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:30.558 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:24:30.559 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:30.565+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:30.660 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:30.663+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:30.684 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:30.684 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:30.698 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:24:30.761 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:30.763+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:30.813 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:24:30.814 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.13 to 1 (10000) 2024-04-26T01:24:30.821 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:30.821 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:179: done (2 left) 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:180: read oid 4 snap -1 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:180: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:181: rollback oid 26 current snap is 12 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 26 to 6 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:174: finishing copy_from to smithi18128752-17 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 29 (ObjNum 69 snap 6 seq_num 69) dirty exists 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:178: finishing copy_from racing read to smithi18128752-27 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:178: finishing copy_from to smithi18128752-27 2024-04-26T01:24:30.822 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 27 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-04-26T01:24:30.823 INFO:tasks.rados.rados.0.smithi181.stdout:174: done (3 left) 2024-04-26T01:24:30.823 INFO:tasks.rados.rados.0.smithi181.stdout:178: done (2 left) 2024-04-26T01:24:30.823 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 16 current snap is 12 2024-04-26T01:24:30.823 INFO:tasks.rados.rados.0.smithi181.stdout:182: seq_num 77 ranges {2768896=524288,3293184=524288,3817472=245760} 2024-04-26T01:24:30.843 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-26T01:24:30.844 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:24:30.875 INFO:tasks.rados.rados.0.smithi181.stdout:182: writing smithi18128752-16 from 2768896 to 3293184 tid 1 2024-04-26T01:24:30.882 INFO:tasks.rados.rados.0.smithi181.stdout:182: writing smithi18128752-16 from 3293184 to 3817472 tid 2 2024-04-26T01:24:30.884 INFO:tasks.rados.rados.0.smithi181.stdout:182: writing smithi18128752-16 from 3817472 to 4063232 tid 3 2024-04-26T01:24:30.884 INFO:tasks.rados.rados.0.smithi181.stdout:183: read oid 28 snap -1 2024-04-26T01:24:30.884 INFO:tasks.rados.rados.0.smithi181.stdout:183: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:24:30.884 INFO:tasks.rados.rados.0.smithi181.stdout:184: rollback oid 40 current snap is 12 2024-04-26T01:24:30.885 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 40 to 2 2024-04-26T01:24:30.885 INFO:tasks.rados.rados.0.smithi181.stdout:185: delete oid 27 current snap is 12 2024-04-26T01:24:30.890 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:30.891+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:30.963 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:30.963+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:31.009 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:31.009 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:31.025 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:24:31.063 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:31.069+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:31.164 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:31.169+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:31.165 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:31.165 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:31.179 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:24:31.266 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:31.267+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:31.308 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:31.308 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:31.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:24:31.327 INFO:tasks.rados.rados.0.smithi181.stdout:183: expect (ObjNum 27 snap 0 seq_num 27) 2024-04-26T01:24:31.361 INFO:tasks.rados.rados.0.smithi181.stdout:181: finishing rollback tid 0 to smithi18128752-26 2024-04-26T01:24:31.367 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:31.369+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:31.427 INFO:tasks.rados.rados.0.smithi181.stdout:180: expect (ObjNum 3 snap 0 seq_num 3) 2024-04-26T01:24:31.436 INFO:tasks.rados.rados.0.smithi181.stdout:181: finishing rollback tid 1 to smithi18128752-26 2024-04-26T01:24:31.437 INFO:tasks.rados.rados.0.smithi181.stdout:181: finishing rollback tid 2 to smithi18128752-26 2024-04-26T01:24:31.437 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 26 v 38 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-04-26T01:24:31.450 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:31.450 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:31.464 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:24:31.467 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:31.467+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:31.568 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:31.571+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:31.590 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:31.590 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:31.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:24:31.639 INFO:tasks.rados.rados.0.smithi181.stdout:184: finishing rollback tid 0 to smithi18128752-40 2024-04-26T01:24:31.639 INFO:tasks.rados.rados.0.smithi181.stdout:184: finishing rollback tid 1 to smithi18128752-40 2024-04-26T01:24:31.669 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:31.671+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:31.693 INFO:tasks.rados.rados.0.smithi181.stdout:182: finishing write tid 1 to smithi18128752-16 2024-04-26T01:24:31.693 INFO:tasks.rados.rados.0.smithi181.stdout:182: finishing write tid 2 to smithi18128752-16 2024-04-26T01:24:31.694 INFO:tasks.rados.rados.0.smithi181.stdout:182: finishing write tid 3 to smithi18128752-16 2024-04-26T01:24:31.694 INFO:tasks.rados.rados.0.smithi181.stdout:182: finishing write tid 4 to smithi18128752-16 2024-04-26T01:24:31.731 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:31.732 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:31.740 INFO:tasks.rados.rados.0.smithi181.stdout:184: finishing rollback tid 2 to smithi18128752-40 2024-04-26T01:24:31.740 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 46 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T01:24:31.741 INFO:tasks.rados.rados.0.smithi181.stdout:180: done (5 left) 2024-04-26T01:24:31.741 INFO:tasks.rados.rados.0.smithi181.stdout:181: done (4 left) 2024-04-26T01:24:31.741 INFO:tasks.rados.rados.0.smithi181.stdout:183: done (3 left) 2024-04-26T01:24:31.741 INFO:tasks.rados.rados.0.smithi181.stdout:184: done (2 left) 2024-04-26T01:24:31.741 INFO:tasks.rados.rados.0.smithi181.stdout:185: done (1 left) 2024-04-26T01:24:31.741 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 48 current snap is 12 2024-04-26T01:24:31.742 INFO:tasks.rados.rados.0.smithi181.stdout:186: seq_num 79 ranges {1900544=573440,2473984=425984,2899968=344064} 2024-04-26T01:24:31.746 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:24:31.768 INFO:tasks.rados.rados.0.smithi181.stdout:186: writing smithi18128752-48 from 1900544 to 2473984 tid 1 2024-04-26T01:24:31.769 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:31.773+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:31.773 INFO:tasks.rados.rados.0.smithi181.stdout:186: writing smithi18128752-48 from 2473984 to 2899968 tid 2 2024-04-26T01:24:31.777 INFO:tasks.rados.rados.0.smithi181.stdout:186: writing smithi18128752-48 from 2899968 to 3244032 tid 3 2024-04-26T01:24:31.777 INFO:tasks.rados.rados.0.smithi181.stdout:187: snap_create 2024-04-26T01:24:31.822 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.14 to 1 (10000) 2024-04-26T01:24:31.828 INFO:tasks.rados.rados.0.smithi181.stdout:187: done (2 left) 2024-04-26T01:24:31.828 INFO:tasks.rados.rados.0.smithi181.stdout:188: snap_remove snap 6 2024-04-26T01:24:31.847 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-26T01:24:31.872 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:31.872 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:31.890 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:24:31.891 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:31.895+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:31.971 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:31.973+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:32.009 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:32.010 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:32.024 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:32.074 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:32.075+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:32.154 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:32.154 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:32.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:32.173 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:32.175+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:32.274 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:32.277+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:32.346 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:32.346 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:32.353 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:32.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:32.375 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:32.377+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:32.476 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:32.475+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:32.483 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:32.483 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:32.498 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:32.512 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:32.513 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:32.532 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:32.532 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:32.555 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:32.556 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:32.577 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:32.579+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:32.579 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:32.579 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:32.600 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:32.600 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:32.622 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:32.623 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:32.645 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:32.645 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:32.669 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:32.670 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:32.670 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:32.670 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:32.678 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:32.679+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:32.686 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:32.690 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:32.690 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:32.711 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:32.711 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:32.734 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:32.735 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:32.759 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:32.759 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:32.779 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:32.779+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:32.781 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:32.781 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:32.802 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:32.803 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:32.824 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:32.824 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:32.825 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.15 to 1 (10000) 2024-04-26T01:24:32.831 INFO:tasks.rados.rados.0.smithi181.stdout:188: done (2 left) 2024-04-26T01:24:32.831 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 45 current snap is 13 2024-04-26T01:24:32.831 INFO:tasks.rados.rados.0.smithi181.stdout:182: finishing write tid 5 to smithi18128752-16 2024-04-26T01:24:32.831 INFO:tasks.rados.rados.0.smithi181.stdout:182: oid 16 updating version 0 to 37 2024-04-26T01:24:32.832 INFO:tasks.rados.rados.0.smithi181.stdout:182: oid 16 version 37 is already newer than 36 2024-04-26T01:24:32.832 INFO:tasks.rados.rados.0.smithi181.stdout:182: oid 16 version 37 is already newer than 33 2024-04-26T01:24:32.832 INFO:tasks.rados.rados.0.smithi181.stdout:182: oid 16 version 37 is already newer than 32 2024-04-26T01:24:32.832 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 37 (ObjNum 77 snap 12 seq_num 77) dirty exists 2024-04-26T01:24:32.832 INFO:tasks.rados.rados.0.smithi181.stdout:182: left oid 16 (ObjNum 77 snap 12 seq_num 77) 2024-04-26T01:24:32.832 INFO:tasks.rados.rados.0.smithi181.stdout:189: seq_num 80 ranges {753664=589824,1343488=573440,1916928=196608} 2024-04-26T01:24:32.843 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:24:32.849 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T01:24:32.850 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:24:32.850 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:32.851 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:32.851 INFO:tasks.rados.rados.0.smithi181.stdout:189: writing smithi18128752-45 from 753664 to 1343488 tid 1 2024-04-26T01:24:32.861 INFO:tasks.rados.rados.0.smithi181.stdout:189: writing smithi18128752-45 from 1343488 to 1916928 tid 2 2024-04-26T01:24:32.864 INFO:tasks.rados.rados.0.smithi181.stdout:189: writing smithi18128752-45 from 1916928 to 2113536 tid 3 2024-04-26T01:24:32.864 INFO:tasks.rados.rados.0.smithi181.stdout:182: done (2 left) 2024-04-26T01:24:32.864 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 20 current snap is 13 2024-04-26T01:24:32.864 INFO:tasks.rados.rados.0.smithi181.stdout:190: seq_num 81 ranges {81920=606208,688128=720896,1409024=49152} 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 41.237910388000003, 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40356932699999998, 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.867 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 37 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630284+0000", 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4700000000000001e-07 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630292+0000", 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3669999999999999e-06 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630302+0000", 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0115999999999999e-05 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.630310+0000", 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.2029999999999992e-06 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735212+0000", 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104902254 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735342+0000", 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012980500000000001 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.735542+0000", 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020034 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:51.736506+0000", 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00096400499999999998 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033692+0000", 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29718516499999997 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033783+0000", 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1377000000000003e-05 2024-04-26T01:24:32.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:52.033853+0000", 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9948000000000002e-05 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:84 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 37.936804103, 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40787895499999999, 2024-04-26T01:24:32.873 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 84 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931390+0000", 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931392+0000", 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7659999999999999e-06 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931546+0000", 2024-04-26T01:24:32.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015429199999999999 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931571+0000", 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5381999999999999e-05 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931597+0000", 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5616e-05 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934003+0000", 2024-04-26T01:24:32.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0024058180000000001 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934128+0000", 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012505200000000001 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934351+0000", 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000222912 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935791+0000", 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014398799999999999 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339098+0000", 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40330716900000002 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339195+0000", 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7676999999999993e-05 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339269+0000", 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3391000000000003e-05 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:85 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 37.936561898000001, 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.407895061, 2024-04-26T01:24:32.879 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 85 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931632+0000", 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931634+0000", 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4330000000000001e-06 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931646+0000", 2024-04-26T01:24:32.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2622999999999999e-05 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931665+0000", 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8669999999999999e-05 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931699+0000", 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4112000000000001e-05 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.882 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.934956+0000", 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0032571119999999999 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935079+0000", 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122483 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.935269+0000", 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019042700000000001 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.936573+0000", 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013043340000000001 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339389+0000", 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.402815335 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339468+0000", 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9758000000000004e-05 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339527+0000", 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8773999999999997e-05 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:86 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 37.936466904, 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40806487499999999, 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.885 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 86 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931727+0000", 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931728+0000", 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3030000000000001e-06 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931739+0000", 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.046e-05 2024-04-26T01:24:32.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:54.931758+0000", 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8842e-05 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.031855+0000", 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100097588 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032039+0000", 2024-04-26T01:24:32.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000184141 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032212+0000", 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000172747 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032428+0000", 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00021604 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.889 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.135784+0000", 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10335620700000001 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339636+0000", 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20385123199999999 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339735+0000", 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9036999999999994e-05 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.339792+0000", 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7278000000000003e-05 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 37.836094971999998, 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40781067500000001, 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.891 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 88 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032099+0000", 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032101+0000", 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7630000000000001e-06 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032115+0000", 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4375999999999999e-05 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032137+0000", 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1356e-05 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.032158+0000", 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1253000000000001e-05 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134605+0000", 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102446813 2024-04-26T01:24:32.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134733+0000", 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012879999999999999 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.134935+0000", 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020153599999999999 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.136395+0000", 2024-04-26T01:24:32.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014603089999999999 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439763+0000", 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30336767999999997 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439846+0000", 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3430999999999995e-05 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:55.439910+0000", 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3357999999999999e-05 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462686929999997, 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41242393999999999, 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.897 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 110 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405507+0000", 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405508+0000", 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3699999999999999e-07 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405515+0000", 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.669e-06 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405525+0000", 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0657e-05 2024-04-26T01:24:32.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405533+0000", 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0160000000000008e-06 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414083+0000", 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0085497569999999998 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414207+0000", 2024-04-26T01:24:32.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012396999999999999 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414400+0000", 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019254 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520650+0000", 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10625069299999999 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817668+0000", 2024-04-26T01:24:32.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29701819000000002 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817820+0000", 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000151075 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.817931+0000", 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011143599999999999 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462648244999997, 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51275573399999996, 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 111 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405546+0000", 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6700000000000003e-07 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405552+0000", 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0930000000000002e-06 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405561+0000", 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5030000000000003e-06 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405572+0000", 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0723000000000001e-05 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.414943+0000", 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0093715910000000003 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.415187+0000", 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024357900000000001 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.415560+0000", 2024-04-26T01:24:32.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00037288799999999998 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520666+0000", 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105105713 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918150+0000", 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.39748471200000002 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918236+0000", 2024-04-26T01:24:32.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5780999999999997e-05 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918301+0000", 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5483999999999998e-05 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:32.908 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462610005999998, 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51318251699999995, 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 112 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:32.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405584+0000", 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405585+0000", 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5300000000000003e-07 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405743+0000", 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015815499999999999 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405757+0000", 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3997e-05 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405767+0000", 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6260000000000007e-06 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416357+0000", 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.010590268999999999 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416478+0000", 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012163300000000001 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.416669+0000", 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019071999999999999 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522454+0000", 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105784641 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918478+0000", 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.396024563 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918641+0000", 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016244100000000001 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.918767+0000", 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012571899999999999 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462409442000002, 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61300901299999999, 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.914 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 113 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405786+0000", 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5600000000000004e-07 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405803+0000", 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7666e-05 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405814+0000", 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0747000000000001e-05 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405822+0000", 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8329999999999994e-06 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417320+0000", 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011497772999999999 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417564+0000", 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000244842 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.418017+0000", 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045273500000000001 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522469+0000", 2024-04-26T01:24:32.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104451424 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018558+0000", 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.496089267 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018655+0000", 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7194000000000005e-05 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018794+0000", 2024-04-26T01:24:32.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013857600000000001 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462359210999999, 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61322511700000004, 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.920 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 114 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:32.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1699999999999997e-07 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405842+0000", 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8530000000000003e-06 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405852+0000", 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6830000000000003e-06 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405860+0000", 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6159999999999996e-06 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.923 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.518995+0000", 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11313486 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519240+0000", 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000245593 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519639+0000", 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039928400000000002 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524261+0000", 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046215249999999996 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018869+0000", 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49460855799999998 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018947+0000", 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7742000000000002e-05 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019060+0000", 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112686 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:32.926 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462318428000003, 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61349101800000005, 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.16e-07 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406016+0000", 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013951499999999999 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406029+0000", 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2819999999999999e-05 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406037+0000", 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6470000000000004e-06 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520774+0000", 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.114736465 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520941+0000", 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016684800000000001 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.521147+0000", 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020602000000000001 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524283+0000", 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003136453 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019192+0000", 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.494908399 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019283+0000", 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1893999999999998e-05 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019367+0000", 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3140999999999996e-05 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462138447999997, 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61359576599999999, 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.935 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6400000000000001e-07 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406062+0000", 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5160000000000002e-06 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406072+0000", 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.91e-06 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406080+0000", 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7030000000000006e-06 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522858+0000", 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116778593 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523093+0000", 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000234763 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523479+0000", 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000385831 2024-04-26T01:24:32.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525176+0000", 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016968619999999999 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019468+0000", 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49429284899999998 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019546+0000", 2024-04-26T01:24:32.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7553999999999996e-05 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019651+0000", 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000105321 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 36.462102522000002, 2024-04-26T01:24:32.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61381219799999998, 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 117 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406092+0000", 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6700000000000003e-07 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406100+0000", 2024-04-26T01:24:32.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4560000000000003e-06 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406109+0000", 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4590000000000004e-06 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.506278+0000", 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10016914 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524347+0000", 2024-04-26T01:24:32.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018068623999999998 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524495+0000", 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148276 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524702+0000", 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020681900000000001 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525680+0000", 2024-04-26T01:24:32.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00097787599999999992 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019759+0000", 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49407962300000002 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019840+0000", 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0563999999999996e-05 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019904+0000", 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3694000000000004e-05 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:189 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:00.690634+0000", 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.177560038000003, 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30989272800000001, 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.947 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 189 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690634+0000", 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690634+0000", 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690635+0000", 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4199999999999995e-07 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690640+0000", 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2140000000000004e-06 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690648+0000", 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1040000000000007e-06 2024-04-26T01:24:32.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.690658+0000", 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7389999999999994e-06 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.789541+0000", 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098883440000000003 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.789695+0000", 2024-04-26T01:24:32.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000153638 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.789927+0000", 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023231400000000001 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:00.791113+0000", 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011859850000000001 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:24:32.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:01.000384+0000", 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20927051599999999 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:01.000464+0000", 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0317999999999999e-05 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:01.000527+0000", 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2817999999999994e-05 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:167 3.2s0 3:42c5d4a2:::smithi18128752-44:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:07.297589+0000", 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 25.570605177000001, 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40316530499999997, 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 167 2024-04-26T01:24:32.953 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297589+0000", 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297589+0000", 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.954 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297592+0000", 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.768e-06 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297606+0000", 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4229999999999999e-05 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297639+0000", 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3188000000000001e-05 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.955 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297655+0000", 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6090999999999999e-05 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.297844+0000", 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018922599999999999 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.298022+0000", 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017731600000000001 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.956 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.406673+0000", 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.108651228 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.408960+0000", 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002286883 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.700582+0000", 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29162229299999998 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.957 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.700754+0000", 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000172082 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:387 3.2s0 3:40bd2c39:::smithi18128752-16:head [append 0~458752] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.231167+0000", 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.6370273560000008, 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.31295971500000003, 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.958 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 387 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.231167+0000", 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.959 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.231167+0000", 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.231169+0000", 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3970000000000001e-06 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.231864+0000", 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000695029 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.960 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.231905+0000", 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1437999999999999e-05 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.231929+0000", 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3349e-05 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.232129+0000", 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020021199999999999 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.961 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op must wait for map", 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.232152+0000", 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3161999999999999e-05 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.335767+0000", 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103614365 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.335893+0000", 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012677000000000001 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.338394+0000", 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002501124 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441905+0000", 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103510852 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.543932+0000", 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102026613 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.544126+0000", 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019440399999999999 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:388 3.2s0 3:40bd2c39:::smithi18128752-16:head [append 0~475136] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.236887+0000", 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.6313065929999997, 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51025956500000003, 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.964 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 388 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.236887+0000", 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.965 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.236887+0000", 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.236890+0000", 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2730000000000001e-06 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.237537+0000", 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064713499999999996 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.966 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.237568+0000", 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1405000000000002e-05 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.237584+0000", 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5549999999999999e-05 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.237757+0000", 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017298699999999999 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.967 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.237777+0000", 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0009000000000001e-05 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.340570+0000", 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10279345300000001 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.340729+0000", 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000158634 2024-04-26T01:24:32.968 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342115+0000", 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013857590000000001 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441907+0000", 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.099792887999999996 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.969 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.747015+0000", 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30510716700000001 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.747147+0000", 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013230499999999999 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:389 3.2s0 3:40bd2c39:::smithi18128752-16:head [append 0~294912] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:32.970 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240184+0000", 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.6280101079999998, 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.607559242, 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 389 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.971 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240184+0000", 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240184+0000", 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240186+0000", 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0360000000000001e-06 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.972 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240598+0000", 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041248800000000003 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240629+0000", 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0621999999999999e-05 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240644+0000", 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5396e-05 2024-04-26T01:24:32.973 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240755+0000", 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011019900000000001 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240766+0000", 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1087000000000001e-05 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.440598+0000", 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19983246599999999 2024-04-26T01:24:32.974 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.440691+0000", 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2369999999999998e-05 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441366+0000", 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067539800000000001 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.746371+0000", 2024-04-26T01:24:32.975 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30500468200000003 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.847590+0000", 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10121896800000001 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.847743+0000", 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015353 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.976 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:390 3.2s0 3:40bd2c39:::smithi18128752-16:head [setxattr _header (56) in=7b] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240675+0000", 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.6275191469999992, 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.607429153, 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 390 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.977 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240675+0000", 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240675+0000", 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240676+0000", 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0160000000000001e-06 2024-04-26T01:24:32.978 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240681+0000", 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5840000000000002e-06 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240693+0000", 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1552999999999999e-05 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240702+0000", 2024-04-26T01:24:32.979 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.772e-06 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240860+0000", 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015798099999999999 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240880+0000", 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0222000000000001e-05 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.980 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.544242+0000", 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30336164199999999 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.544417+0000", 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000175773 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.545240+0000", 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00082237499999999997 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.981 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.746503+0000", 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201262949 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848038+0000", 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101535325 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848104+0000", 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5961000000000001e-05 2024-04-26T01:24:32.982 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:391 3.2s0 3:40bd2c39:::smithi18128752-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240751+0000", 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.627443177, 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.811133887, 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:32.983 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 391 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240751+0000", 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240751+0000", 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.984 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240752+0000", 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240756+0000", 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1459999999999998e-06 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240768+0000", 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2456999999999999e-05 2024-04-26T01:24:32.985 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240777+0000", 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8859999999999993e-06 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240928+0000", 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015100399999999999 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240937+0000", 2024-04-26T01:24:32.986 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7730000000000005e-06 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.849959+0000", 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60902194200000004 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.850122+0000", 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016275999999999999 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:32.987 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.051885+0000", 2024-04-26T01:24:32.988 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201762952 2024-04-26T01:24:32.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.988 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.988 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:32.988 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:32.988 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:32.991 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:32.992 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:32.987+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:32.992 INFO:tasks.rados.rados.0.smithi181.stdout:190: writing smithi18128752-20 from 81920 to 688128 tid 1 2024-04-26T01:24:32.992 INFO:tasks.rados.rados.0.smithi181.stdout:190: writing smithi18128752-20 from 688128 to 1409024 tid 2 2024-04-26T01:24:32.992 INFO:tasks.rados.rados.0.smithi181.stdout:190: writing smithi18128752-20 from 1409024 to 1458176 tid 3 2024-04-26T01:24:32.992 INFO:tasks.rados.rados.0.smithi181.stdout:191: setattr oid 4 current snap is 13 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:186: finishing write tid 1 to smithi18128752-48 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:186: finishing write tid 2 to smithi18128752-48 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:186: finishing write tid 3 to smithi18128752-48 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:186: finishing write tid 4 to smithi18128752-48 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:192: read oid 44 snap 2 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:192: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 9 current snap is 13 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:193: seq_num 82 ranges {147456=622592,770048=475136,1245184=294912} 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:193: writing smithi18128752-9 from 147456 to 770048 tid 1 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:193: writing smithi18128752-9 from 770048 to 1245184 tid 2 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:193: writing smithi18128752-9 from 1245184 to 1540096 tid 3 2024-04-26T01:24:32.993 INFO:tasks.rados.rados.0.smithi181.stdout:194: snap_create 2024-04-26T01:24:33.090 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:33.091+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:33.124 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:33.124 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:33.125 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:33.125 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:33.138 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:33.161 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:24:33.162 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:24:33.182 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-04-26T01:24:32.831168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":42648096,"num_objects":88,"num_object_clones":36,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":5,"num_read":399,"num_read_kb":45811,"num_write":495,"num_write_kb":52580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":21942816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":523,"ondisk_log_size":523,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":462624,"kb_used_data":78064,"kb_used_omap":20,"kb_used_meta":384363,"kb_avail":1226371296,"statfs":{"total":1256277934080,"available":1255804207104,"internally_reserved":0,"allocated":79937536,"data_stored":101146938,"data_compressed":1355988,"data_compressed_allocated":1871872,"data_compressed_original":4616192,"omap_allocated":20741,"internal_metadata":393588475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1081344,"num_objects":2,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1585,"num_write":3,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.058085"},"pg_stats":[{"pgid":"3.f","version":"34'4","reported_seq":139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.934209+0000","last_change":"2024-04-26T01:24:27.934570+0000","last_active":"2024-04-26T01:24:27.934209+0000","last_peered":"2024-04-26T01:24:27.934209+0000","last_clean":"2024-04-26T01:24:27.934209+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:24:27.934209+0000","last_undegraded":"2024-04-26T01:24:27.934209+0000","last_fullsized":"2024-04-26T01:24:27.934209+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:11.205399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062840599999999997,"stat_sum":{"num_bytes":753664,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1683,"num_write":4,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.e","version":"48'40","reported_seq":100,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.929207+0000","last_change":"2024-04-26T01:24:27.930138+0000","last_active":"2024-04-26T01:24:27.929207+0000","last_peered":"2024-04-26T01:24:27.929207+0000","last_clean":"2024-04-26T01:24:27.929207+0000","last_became_active":"2024-04-26T01:24:16.649834+0000","last_became_peered":"2024-04-26T01:24:16.649834+0000","last_unstale":"2024-04-26T01:24:27.929207+0000","last_undegraded":"2024-04-26T01:24:27.929207+0000","last_fullsized":"2024-04-26T01:24:27.929207+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.264283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015485,"stat_sum":{"num_bytes":3145728,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":22,"num_read_kb":2297,"num_write":20,"num_write_kb":4000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5128192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,14],"acting":[13,9,1,11,8,6,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.d","version":"49'31","reported_seq":176,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.796254+0000","last_change":"2024-04-26T01:24:24.953056+0000","last_active":"2024-04-26T01:24:26.796254+0000","last_peered":"2024-04-26T01:24:26.796254+0000","last_clean":"2024-04-26T01:24:26.796254+0000","last_became_active":"2024-04-26T01:24:16.546214+0000","last_became_peered":"2024-04-26T01:24:16.546214+0000","last_unstale":"2024-04-26T01:24:26.796254+0000","last_undegraded":"2024-04-26T01:24:26.796254+0000","last_fullsized":"2024-04-26T01:24:26.796254+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:23.436774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10123578599999999,"stat_sum":{"num_bytes":3670016,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":5096,"num_write":17,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14,10,0,8,6],"acting":[4,3,14,10,0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"3.c","version":"53'34","reported_seq":170,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.933972+0000","last_change":"2024-04-26T01:24:27.934343+0000","last_active":"2024-04-26T01:24:27.933972+0000","last_peered":"2024-04-26T01:24:27.933972+0000","last_clean":"2024-04-26T01:24:27.933972+0000","last_became_active":"2024-04-26T01:24:05.439203+0000","last_became_peered":"2024-04-26T01:24:05.439203+0000","last_unstale":"2024-04-26T01:24:27.933972+0000","last_undegraded":"2024-04-26T01:24:27.933972+0000","last_fullsized":"2024-04-26T01:24:27.933972+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.163979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059489899999999997,"stat_sum":{"num_bytes":4505600,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":21,"num_read_kb":2408,"num_write":39,"num_write_kb":4800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,14,5],"acting":[11,8,0,10,6,14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.b","version":"48'10","reported_seq":137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.796083+0000","last_change":"2024-04-26T01:24:24.952945+0000","last_active":"2024-04-26T01:24:26.796083+0000","last_peered":"2024-04-26T01:24:26.796083+0000","last_clean":"2024-04-26T01:24:26.796083+0000","last_became_active":"2024-04-26T01:24:05.273930+0000","last_became_peered":"2024-04-26T01:24:05.273930+0000","last_unstale":"2024-04-26T01:24:26.796083+0000","last_undegraded":"2024-04-26T01:24:26.796083+0000","last_fullsized":"2024-04-26T01:24:26.796083+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.245183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10120567499999999,"stat_sum":{"num_bytes":81920,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":573,"num_write":23,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,9,6,3,13],"acting":[4,0,1,9,6,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"3.a","version":"44'15","reported_seq":149,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-26T01:24:31.088741+0000","last_change":"2024-04-26T01:24:31.089971+0000","last_active":"2024-04-26T01:24:31.088741+0000","last_peered":"2024-04-26T01:24:31.088741+0000","last_clean":"2024-04-26T01:24:31.088741+0000","last_became_active":"2024-04-26T01:24:01.227642+0000","last_became_peered":"2024-04-26T01:24:01.227642+0000","last_unstale":"2024-04-26T01:24:31.088741+0000","last_undegraded":"2024-04-26T01:24:31.088741+0000","last_fullsized":"2024-04-26T01:24:31.088741+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.139748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00151727,"stat_sum":{"num_bytes":65536,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":25,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1,8,14,6,11],"acting":[3,9,1,8,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9","length":"5"}]},{"pgid":"3.9","version":"53'22","reported_seq":141,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.817622+0000","last_change":"2024-04-26T01:24:25.154830+0000","last_active":"2024-04-26T01:24:26.817622+0000","last_peered":"2024-04-26T01:24:26.817622+0000","last_clean":"2024-04-26T01:24:26.817622+0000","last_became_active":"2024-04-26T01:24:01.291346+0000","last_became_peered":"2024-04-26T01:24:01.291346+0000","last_unstale":"2024-04-26T01:24:26.817622+0000","last_undegraded":"2024-04-26T01:24:26.817622+0000","last_fullsized":"2024-04-26T01:24:26.817622+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.912177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27000373100000002,"stat_sum":{"num_bytes":1540096,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":17,"num_read_kb":1013,"num_write":12,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"3.8","version":"59'57","reported_seq":237,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T01:24:32.081208+0000","last_change":"2024-04-26T01:24:31.137363+0000","last_active":"2024-04-26T01:24:32.081208+0000","last_peered":"2024-04-26T01:24:32.081208+0000","last_clean":"2024-04-26T01:24:32.081208+0000","last_became_active":"2024-04-26T01:24:16.447854+0000","last_became_peered":"2024-04-26T01:24:16.447854+0000","last_unstale":"2024-04-26T01:24:32.081208+0000","last_undegraded":"2024-04-26T01:24:32.081208+0000","last_fullsized":"2024-04-26T01:24:32.081208+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:24.524246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.214096435,"stat_sum":{"num_bytes":7979008,"num_objects":12,"num_object_clones":7,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":33,"num_read_kb":11229,"num_write":51,"num_write_kb":8992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4161536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1,9,14,15,6],"acting":[5,0,1,9,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"9","length":"5"}]},{"pgid":"1.0","version":"34'157","reported_seq":281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.028041+0000","last_change":"2024-04-26T01:24:19.157462+0000","last_active":"2024-04-26T01:24:28.028041+0000","last_peered":"2024-04-26T01:24:28.028041+0000","last_clean":"2024-04-26T01:24:28.028041+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:24:28.028041+0000","last_undegraded":"2024-04-26T01:24:28.028041+0000","last_fullsized":"2024-04-26T01:24:28.028041+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:23:48.870422+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:54.023472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T01:24:31.881036+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:24:31.881036+0000","last_peered":"2024-04-26T01:24:31.881036+0000","last_clean":"2024-04-26T01:24:31.881036+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:24:31.881036+0000","last_undegraded":"2024-04-26T01:24:31.881036+0000","last_fullsized":"2024-04-26T01:24:31.881036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'31","reported_seq":171,"reported_epoch":56,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T01:24:28.133596+0000","last_change":"2024-04-26T01:24:27.929318+0000","last_active":"2024-04-26T01:24:28.133596+0000","last_peered":"2024-04-26T01:24:28.133596+0000","last_clean":"2024-04-26T01:24:28.133596+0000","last_became_active":"2024-04-26T01:24:16.348216+0000","last_became_peered":"2024-04-26T01:24:16.348216+0000","last_unstale":"2024-04-26T01:24:28.133596+0000","last_undegraded":"2024-04-26T01:24:28.133596+0000","last_fullsized":"2024-04-26T01:24:28.133596+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.592546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20613052000000001,"stat_sum":{"num_bytes":3178496,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":342,"num_write":17,"num_write_kb":3824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-26T01:24:30.989673+0000","last_change":"2024-04-26T01:23:52.211699+0000","last_active":"2024-04-26T01:24:30.989673+0000","last_peered":"2024-04-26T01:24:30.989673+0000","last_clean":"2024-04-26T01:24:30.989673+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:24:30.989673+0000","last_undegraded":"2024-04-26T01:24:30.989673+0000","last_fullsized":"2024-04-26T01:24:30.989673+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:51.317271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"45'14","reported_seq":149,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.896080+0000","last_change":"2024-04-26T01:24:25.053594+0000","last_active":"2024-04-26T01:24:26.896080+0000","last_peered":"2024-04-26T01:24:26.896080+0000","last_clean":"2024-04-26T01:24:26.896080+0000","last_became_active":"2024-04-26T01:24:16.442216+0000","last_became_peered":"2024-04-26T01:24:16.442216+0000","last_unstale":"2024-04-26T01:24:26.896080+0000","last_undegraded":"2024-04-26T01:24:26.896080+0000","last_fullsized":"2024-04-26T01:24:26.896080+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.617908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101877333,"stat_sum":{"num_bytes":2621440,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4406,"num_write":10,"num_write_kb":3312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,14,13],"acting":[4,11,9,5,10,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.928015+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:24:27.928015+0000","last_peered":"2024-04-26T01:24:27.928015+0000","last_clean":"2024-04-26T01:24:27.928015+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:24:27.928015+0000","last_undegraded":"2024-04-26T01:24:27.928015+0000","last_fullsized":"2024-04-26T01:24:27.928015+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"55'34","reported_seq":186,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.228083+0000","last_change":"2024-04-26T01:24:24.844011+0000","last_active":"2024-04-26T01:24:27.228083+0000","last_peered":"2024-04-26T01:24:27.228083+0000","last_clean":"2024-04-26T01:24:27.228083+0000","last_became_active":"2024-04-26T01:24:16.549243+0000","last_became_peered":"2024-04-26T01:24:16.549243+0000","last_unstale":"2024-04-26T01:24:27.228083+0000","last_undegraded":"2024-04-26T01:24:27.228083+0000","last_fullsized":"2024-04-26T01:24:27.228083+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:24.413010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017168909999999999,"stat_sum":{"num_bytes":6176768,"num_objects":12,"num_object_clones":4,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":31,"num_read_kb":5342,"num_write":28,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-26T01:24:31.188590+0000","last_change":"2024-04-26T01:24:05.176463+0000","last_active":"2024-04-26T01:24:31.188590+0000","last_peered":"2024-04-26T01:24:31.188590+0000","last_clean":"2024-04-26T01:24:31.188590+0000","last_became_active":"2024-04-26T01:24:05.175683+0000","last_became_peered":"2024-04-26T01:24:05.175683+0000","last_unstale":"2024-04-26T01:24:31.188590+0000","last_undegraded":"2024-04-26T01:24:31.188590+0000","last_fullsized":"2024-04-26T01:24:31.188590+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:57.025465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"35'2","reported_seq":140,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.029043+0000","last_change":"2024-04-26T01:24:28.029612+0000","last_active":"2024-04-26T01:24:28.029043+0000","last_peered":"2024-04-26T01:24:28.029043+0000","last_clean":"2024-04-26T01:24:28.029043+0000","last_became_active":"2024-04-26T01:24:16.644086+0000","last_became_peered":"2024-04-26T01:24:16.644086+0000","last_unstale":"2024-04-26T01:24:28.029043+0000","last_undegraded":"2024-04-26T01:24:28.029043+0000","last_fullsized":"2024-04-26T01:24:28.029043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:14.994473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091882499999999996,"stat_sum":{"num_bytes":622592,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,4,15,6,5,3],"acting":[13,9,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.5","version":"0'0","reported_seq":128,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.823583+0000","last_change":"2024-04-26T01:24:24.884359+0000","last_active":"2024-04-26T01:24:26.823583+0000","last_peered":"2024-04-26T01:24:26.823583+0000","last_clean":"2024-04-26T01:24:26.823583+0000","last_became_active":"2024-04-26T01:24:16.447258+0000","last_became_peered":"2024-04-26T01:24:16.447258+0000","last_unstale":"2024-04-26T01:24:26.823583+0000","last_undegraded":"2024-04-26T01:24:26.823583+0000","last_fullsized":"2024-04-26T01:24:26.823583+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:00.481418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001182701,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,1,0,6],"acting":[15,8,4,13,1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":66,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.795950+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:24:26.795950+0000","last_peered":"2024-04-26T01:24:26.795950+0000","last_clean":"2024-04-26T01:24:26.795950+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:24:26.795950+0000","last_undegraded":"2024-04-26T01:24:26.795950+0000","last_fullsized":"2024-04-26T01:24:26.795950+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"41'12","reported_seq":166,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.776588+0000","last_change":"2024-04-26T01:24:28.125764+0000","last_active":"2024-04-26T01:24:28.776588+0000","last_peered":"2024-04-26T01:24:28.776588+0000","last_clean":"2024-04-26T01:24:28.776588+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:24:28.776588+0000","last_undegraded":"2024-04-26T01:24:28.776588+0000","last_fullsized":"2024-04-26T01:24:28.776588+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:21.162880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011626659999999999,"stat_sum":{"num_bytes":1490944,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":6103,"num_write":8,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"9","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.923352+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:24:26.923352+0000","last_peered":"2024-04-26T01:24:26.923352+0000","last_clean":"2024-04-26T01:24:26.923352+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:24:26.923352+0000","last_undegraded":"2024-04-26T01:24:26.923352+0000","last_fullsized":"2024-04-26T01:24:26.923352+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"55'35","reported_seq":164,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.336662+0000","last_change":"2024-04-26T01:24:24.852327+0000","last_active":"2024-04-26T01:24:27.336662+0000","last_peered":"2024-04-26T01:24:27.336662+0000","last_clean":"2024-04-26T01:24:27.336662+0000","last_became_active":"2024-04-26T01:24:01.091412+0000","last_became_peered":"2024-04-26T01:24:01.091412+0000","last_unstale":"2024-04-26T01:24:27.336662+0000","last_undegraded":"2024-04-26T01:24:27.336662+0000","last_fullsized":"2024-04-26T01:24:27.336662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.458801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089147,"stat_sum":{"num_bytes":3391488,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":32,"num_read_kb":5195,"num_write":19,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.775393+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:24:28.775393+0000","last_peered":"2024-04-26T01:24:28.775393+0000","last_clean":"2024-04-26T01:24:28.775393+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:24:28.775393+0000","last_undegraded":"2024-04-26T01:24:28.775393+0000","last_fullsized":"2024-04-26T01:24:28.775393+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:10.471688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"55'25","reported_seq":160,"reported_epoch":56,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T01:24:27.804697+0000","last_change":"2024-04-26T01:24:27.804737+0000","last_active":"2024-04-26T01:24:27.804697+0000","last_peered":"2024-04-26T01:24:27.804697+0000","last_clean":"2024-04-26T01:24:27.804697+0000","last_became_active":"2024-04-26T01:24:05.252337+0000","last_became_peered":"2024-04-26T01:24:05.252337+0000","last_unstale":"2024-04-26T01:24:27.804697+0000","last_undegraded":"2024-04-26T01:24:27.804697+0000","last_fullsized":"2024-04-26T01:24:27.804697+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:07.504657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20108716700000001,"stat_sum":{"num_bytes":2834432,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":17,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,11,6,9],"acting":[0,13,1,15,11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.823363+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:24:26.823363+0000","last_peered":"2024-04-26T01:24:26.823363+0000","last_clean":"2024-04-26T01:24:26.823363+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:24:26.823363+0000","last_undegraded":"2024-04-26T01:24:26.823363+0000","last_fullsized":"2024-04-26T01:24:26.823363+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":42057728,"num_objects":86,"num_object_clones":36,"num_object_copies":602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":5,"num_read":263,"num_read_kb":45693,"num_write":292,"num_write_kb":49120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21413888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77717504,"data_stored":100634624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":112,"acting":112,"num_store_stats":15},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":0,"seq":94489280524,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":5124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":5246976,"data_stored":6049050,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280523,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37936,"kb_used_data":6756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333904,"statfs":{"total":96636764160,"available":96597917696,"internally_reserved":0,"allocated":6918144,"data_stored":9256218,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34196,"kb_used_data":7176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337644,"statfs":{"total":96636764160,"available":96601747456,"internally_reserved":0,"allocated":7348224,"data_stored":9162010,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313227,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35148,"kb_used_data":8128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336692,"statfs":{"total":96636764160,"available":96600772608,"internally_reserved":0,"allocated":8323072,"data_stored":10308890,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313226,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36936,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334904,"statfs":{"total":96636764160,"available":96598941696,"internally_reserved":0,"allocated":5890048,"data_stored":7720218,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313228,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":2994176,"data_stored":3887246,"data_compressed":143420,"data_compressed_allocated":192512,"data_compressed_original":827392,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313226,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":5000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":5120000,"data_stored":5491994,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313228,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":6888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":7053312,"data_stored":9125486,"data_compressed":109365,"data_compressed_allocated":151552,"data_compressed_original":303104,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313227,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":9004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":9220096,"data_stored":12164378,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":90194313227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313227,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":5904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":6045696,"data_stored":7519514,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313226,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":4592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":4702208,"data_stored":5864730,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313228,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33492,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338348,"statfs":{"total":96636764160,"available":96602468352,"internally_reserved":0,"allocated":6627328,"data_stored":8797466,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313227,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39692,"kb_used_data":4344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332148,"statfs":{"total":96636764160,"available":96596119552,"internally_reserved":0,"allocated":4448256,"data_stored":5799738,"data_compressed":129943,"data_compressed_allocated":176128,"data_compressed_original":782336,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6311936,"data_stored":8568832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6742016,"data_stored":8761344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2613248,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4513792,"data_stored":5091328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6647808,"data_stored":9408512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8663040,"data_stored":11726848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4640768,"data_stored":5648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5488640,"data_stored":7081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096000,"data_stored":5464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6668288,"data_stored":8818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":3809280,"data_stored":4898816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":7737344,"data_stored":9928704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5332992,"data_stored":7368704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:24:33.184 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:24:33.187 INFO:tasks.rados.rados.0.smithi181.stdout:186: finishing write tid 5 to smithi18128752-48 2024-04-26T01:24:33.187 INFO:tasks.rados.rados.0.smithi181.stdout:186: oid 48 updating version 0 to 58 2024-04-26T01:24:33.187 INFO:tasks.rados.rados.0.smithi181.stdout:186: oid 48 version 58 is already newer than 56 2024-04-26T01:24:33.187 INFO:tasks.rados.rados.0.smithi181.stdout:186: oid 48 version 58 is already newer than 57 2024-04-26T01:24:33.188 INFO:tasks.rados.rados.0.smithi181.stdout:186: oid 48 updating version 58 to 59 2024-04-26T01:24:33.188 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 59 (ObjNum 79 snap 12 seq_num 79) dirty exists 2024-04-26T01:24:33.188 INFO:tasks.rados.rados.0.smithi181.stdout:186: left oid 48 (ObjNum 79 snap 12 seq_num 79) 2024-04-26T01:24:33.191 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:33.193+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:33.256 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:33.257 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:33.257 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:33.257 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:33.257 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:33.271 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:33.291 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:33.295+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:33.294 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 1 to smithi18128752-20 2024-04-26T01:24:33.296 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 44 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T01:24:33.345 INFO:tasks.rados.rados.0.smithi181.stdout:192: expect (ObjNum 17 snap 0 seq_num 17) 2024-04-26T01:24:33.392 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:33.395+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:33.395 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:33.395 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:33.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:33.395 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:33.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.396 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:146 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:33.396 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:33.396 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.232511898999999, 2024-04-26T01:24:33.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.841318758, 2024-04-26T01:24:33.396 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.396 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 146 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165148+0000", 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6000000000000001e-06 2024-04-26T01:24:33.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165979+0000", 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00083094699999999996 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166013+0000", 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3386000000000002e-05 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166028+0000", 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5480000000000001e-05 2024-04-26T01:24:33.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189275+0000", 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.023246692999999999 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189585+0000", 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031012999999999999 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.190986+0000", 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014009820000000001 2024-04-26T01:24:33.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295619+0000", 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104633299 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005890+0000", 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71027072300000005 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.401 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006464+0000", 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057451799999999999 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:147 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.226272456, 2024-04-26T01:24:33.402 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93684057600000004, 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 147 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.403 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171387+0000", 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4099999999999998e-06 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171984+0000", 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059681399999999998 2024-04-26T01:24:33.404 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172016+0000", 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1779000000000002e-05 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172031+0000", 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5026e-05 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192109+0000", 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.020077514000000001 2024-04-26T01:24:33.405 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192330+0000", 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00022101 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.193866+0000", 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001536239 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295801+0000", 2024-04-26T01:24:33.406 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101934767 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.107290+0000", 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81148932200000001 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108226+0000", 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093569499999999995 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.407 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:148 3.es0 3:734f3d85:::smithi18128752-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.225598604000002, 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.137414704, 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 148 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.408 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172060+0000", 2024-04-26T01:24:33.409 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.513e-06 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172069+0000", 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0930000000000003e-06 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172088+0000", 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9303000000000001e-05 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.410 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172103+0000", 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5586e-05 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496369+0000", 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32426532299999999 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496614+0000", 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024487400000000001 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.411 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.497656+0000", 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010425289999999999 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.599811+0000", 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10215491 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.308958+0000", 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70914692000000001 2024-04-26T01:24:33.412 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.309474+0000", 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051565300000000003 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:149 3.es0 3:734f3d85:::smithi18128752-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.225518297000001, 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.54710463, 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.413 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 149 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.414 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172141+0000", 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.39e-06 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172148+0000", 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0600000000000002e-06 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172161+0000", 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3803e-05 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172171+0000", 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4069999999999999e-06 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498185+0000", 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.326014476 2024-04-26T01:24:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498406+0000", 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000220461 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311032+0000", 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81262658200000004 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311366+0000", 2024-04-26T01:24:33.417 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00033330099999999999 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.719244+0000", 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40787814999999999 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:33.418 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.938311955, 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0565503999999999, 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.419 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9999999999999998e-07 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459365+0000", 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8600000000000001e-05 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.420 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459378+0000", 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2989e-05 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459388+0000", 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0710000000000001e-05 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.421 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515199+0000", 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.055811056 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515896+0000", 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069664499999999997 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.422 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.936730440000002, 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54462607299999999, 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.423 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460929+0000", 2024-04-26T01:24:33.424 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.72e-06 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460937+0000", 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.763e-06 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460965+0000", 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8693000000000001e-05 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.425 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460979+0000", 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4072e-05 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005512+0000", 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54453236000000005 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005553+0000", 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1464999999999997e-05 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.426 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.936114967000002, 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54510908400000002, 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.427 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461544+0000", 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.428 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461554+0000", 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3700000000000001e-06 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461582+0000", 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8062e-05 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461596+0000", 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.432e-05 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.429 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006604+0000", 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54500755300000003 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006652+0000", 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8041999999999998e-05 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.430 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.922236768000001, 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0397411919999999, 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.431 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475422+0000", 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475446+0000", 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4428999999999999e-05 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.432 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475478+0000", 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1516000000000001e-05 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.575602+0000", 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100124637 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515084+0000", 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93948142300000004 2024-04-26T01:24:33.433 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515162+0000", 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8213999999999993e-05 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.898235543999998, 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60899775199999995, 2024-04-26T01:24:33.434 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.435 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499424+0000", 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7630000000000001e-06 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499436+0000", 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.201e-05 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499465+0000", 2024-04-26T01:24:33.436 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9142999999999999e-05 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499479+0000", 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4419e-05 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108343+0000", 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60886387600000003 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.437 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108420+0000", 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6540999999999998e-05 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=20, to_read={3:7f6c3760:::smithi18128752-23:head=0,4096,0}, subchunks={3:7f6c3760:::smithi18128752-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.79647009, 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91354819600000003, 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.438 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.439 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.440 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601264+0000", 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6395000000000003e-05 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601293+0000", 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8829000000000001e-05 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514183+0000", 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91289044600000002 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.441 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514736+0000", 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055252600000000002 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.79545718, 2024-04-26T01:24:33.442 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.113522984, 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:33.443 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602202+0000", 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602223+0000", 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1036e-05 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.444 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602251+0000", 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7773e-05 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702368+0000", 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100117531 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715639+0000", 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0132701660000001 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.445 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715723+0000", 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4738000000000001e-05 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.695192879, 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81161888599999998, 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.446 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.447 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702467+0000", 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0370000000000001e-06 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702492+0000", 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5096000000000002e-05 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702519+0000", 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7152e-05 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.448 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702533+0000", 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4464000000000001e-05 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514000+0000", 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81146615499999997 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514084+0000", 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3981999999999997e-05 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.619657317000001, 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83750464400000002, 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.450 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.451 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:33.451 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.451 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.451 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.452 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:33.452 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778002+0000", 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.64e-06 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778026+0000", 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.453 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778058+0000", 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2398999999999999e-05 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778076+0000", 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7408999999999999e-05 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615429+0000", 2024-04-26T01:24:33.454 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83735304799999999 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615505+0000", 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6057999999999996e-05 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=21, to_read={3:7e2e4071:::smithi18128752-29:head=0,4096,0}, subchunks={3:7e2e4071:::smithi18128752-29:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:33.455 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.492468078999998, 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018501300000003, 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.456 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.457 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905272+0000", 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2304000000000003e-05 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905302+0000", 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9850000000000001e-05 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615005+0000", 2024-04-26T01:24:33.458 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70970295900000002 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615374+0000", 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036989999999999999 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.491547480000001, 2024-04-26T01:24:33.459 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60991441499999999, 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:33.460 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906112+0000", 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.176e-06 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906133+0000", 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1135999999999999e-05 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.461 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906161+0000", 2024-04-26T01:24:33.462 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7783000000000001e-05 2024-04-26T01:24:33.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906175+0000", 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3419e-05 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515955+0000", 2024-04-26T01:24:33.464 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60978075899999995 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.516024+0000", 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9141999999999996e-05 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.391503205999999, 2024-04-26T01:24:33.465 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71033649600000004, 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.466 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006156+0000", 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006182+0000", 2024-04-26T01:24:33.467 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6262999999999999e-05 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006213+0000", 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0298999999999999e-05 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006233+0000", 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0169e-05 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.468 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716419+0000", 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018600300000001 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716491+0000", 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2132000000000007e-05 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.469 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.391268353000001, 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71023971900000005, 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006391+0000", 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.471 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006420+0000", 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8932999999999999e-05 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006451+0000", 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1086e-05 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006469+0000", 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8016000000000002e-05 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.472 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716554+0000", 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71008492899999998 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716629+0000", 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4857999999999994e-05 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.473 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:254 3.es0 3:7f6c3760:::smithi18128752-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.200783924, 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71203804900000001, 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:33.474 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 254 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.475 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9299999999999997e-07 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196880+0000", 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2410000000000003e-06 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196890+0000", 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0142999999999999e-05 2024-04-26T01:24:33.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.296984+0000", 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100093949 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303480+0000", 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.006496646 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:33.477 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303606+0000", 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012540699999999999 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706770+0000", 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40316384799999999 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706914+0000", 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000144666 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.478 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.908912+0000", 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20199735599999999 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:257 3.es0 3:7915a690:::smithi18128752-22:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.099128707999999, 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71064416799999996, 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.479 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 257 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.480 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298530+0000", 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7999999999999993e-07 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298537+0000", 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8700000000000003e-06 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.481 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298551+0000", 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4202999999999999e-05 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298559+0000", 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0730000000000005e-06 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.482 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709349+0000", 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41079035699999999 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709598+0000", 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024883099999999998 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.710488+0000", 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00088957200000000004 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.483 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.811226+0000", 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100738614 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.008990+0000", 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19776387300000001 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009173+0000", 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000182795 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:258 3.es0 3:7915a690:::smithi18128752-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.099087361999999, 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.99510092800000005, 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 258 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.485 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298571+0000", 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:24:33.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298585+0000", 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4046000000000001e-05 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298599+0000", 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4192999999999999e-05 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298608+0000", 2024-04-26T01:24:33.487 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6030000000000005e-06 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712320+0000", 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41371240100000001 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712472+0000", 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000151586 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009806+0000", 2024-04-26T01:24:33.488 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.297334565 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.010023+0000", 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000216698 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.293671+0000", 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28364788600000002 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.489 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.490 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:33.492 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:33.492 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 2 to smithi18128752-20 2024-04-26T01:24:33.492 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 3 to smithi18128752-20 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 4 to smithi18128752-20 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing write tid 1 to smithi18128752-45 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing write tid 2 to smithi18128752-45 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing write tid 3 to smithi18128752-45 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:193: finishing write tid 1 to smithi18128752-9 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:193: finishing write tid 2 to smithi18128752-9 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:193: finishing write tid 3 to smithi18128752-9 2024-04-26T01:24:33.493 INFO:tasks.rados.rados.0.smithi181.stdout:193: finishing write tid 4 to smithi18128752-9 2024-04-26T01:24:33.494 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:33.499+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:33.535 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:24:33.536 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:24:33.543 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing write tid 4 to smithi18128752-45 2024-04-26T01:24:33.554 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-04-26T01:24:32.831168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":42648096,"num_objects":88,"num_object_clones":36,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":5,"num_read":399,"num_read_kb":45811,"num_write":495,"num_write_kb":52580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":21942816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":523,"ondisk_log_size":523,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":462624,"kb_used_data":78064,"kb_used_omap":20,"kb_used_meta":384363,"kb_avail":1226371296,"statfs":{"total":1256277934080,"available":1255804207104,"internally_reserved":0,"allocated":79937536,"data_stored":101146938,"data_compressed":1355988,"data_compressed_allocated":1871872,"data_compressed_original":4616192,"omap_allocated":20741,"internal_metadata":393588475},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1081344,"num_objects":2,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":1585,"num_write":3,"num_write_kb":1056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.058085"},"pg_stats":[{"pgid":"3.f","version":"34'4","reported_seq":139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.934209+0000","last_change":"2024-04-26T01:24:27.934570+0000","last_active":"2024-04-26T01:24:27.934209+0000","last_peered":"2024-04-26T01:24:27.934209+0000","last_clean":"2024-04-26T01:24:27.934209+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:24:27.934209+0000","last_undegraded":"2024-04-26T01:24:27.934209+0000","last_fullsized":"2024-04-26T01:24:27.934209+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:11.205399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062840599999999997,"stat_sum":{"num_bytes":753664,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":1683,"num_write":4,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.e","version":"48'40","reported_seq":100,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.929207+0000","last_change":"2024-04-26T01:24:27.930138+0000","last_active":"2024-04-26T01:24:27.929207+0000","last_peered":"2024-04-26T01:24:27.929207+0000","last_clean":"2024-04-26T01:24:27.929207+0000","last_became_active":"2024-04-26T01:24:16.649834+0000","last_became_peered":"2024-04-26T01:24:16.649834+0000","last_unstale":"2024-04-26T01:24:27.929207+0000","last_undegraded":"2024-04-26T01:24:27.929207+0000","last_fullsized":"2024-04-26T01:24:27.929207+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.264283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015485,"stat_sum":{"num_bytes":3145728,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":22,"num_read_kb":2297,"num_write":20,"num_write_kb":4000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5128192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,14],"acting":[13,9,1,11,8,6,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.d","version":"49'31","reported_seq":176,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.796254+0000","last_change":"2024-04-26T01:24:24.953056+0000","last_active":"2024-04-26T01:24:26.796254+0000","last_peered":"2024-04-26T01:24:26.796254+0000","last_clean":"2024-04-26T01:24:26.796254+0000","last_became_active":"2024-04-26T01:24:16.546214+0000","last_became_peered":"2024-04-26T01:24:16.546214+0000","last_unstale":"2024-04-26T01:24:26.796254+0000","last_undegraded":"2024-04-26T01:24:26.796254+0000","last_fullsized":"2024-04-26T01:24:26.796254+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:23.436774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10123578599999999,"stat_sum":{"num_bytes":3670016,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":5096,"num_write":17,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2736128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14,10,0,8,6],"acting":[4,3,14,10,0,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"3.c","version":"53'34","reported_seq":170,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.933972+0000","last_change":"2024-04-26T01:24:27.934343+0000","last_active":"2024-04-26T01:24:27.933972+0000","last_peered":"2024-04-26T01:24:27.933972+0000","last_clean":"2024-04-26T01:24:27.933972+0000","last_became_active":"2024-04-26T01:24:05.439203+0000","last_became_peered":"2024-04-26T01:24:05.439203+0000","last_unstale":"2024-04-26T01:24:27.933972+0000","last_undegraded":"2024-04-26T01:24:27.933972+0000","last_fullsized":"2024-04-26T01:24:27.933972+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.163979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059489899999999997,"stat_sum":{"num_bytes":4505600,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":21,"num_read_kb":2408,"num_write":39,"num_write_kb":4800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,14,5],"acting":[11,8,0,10,6,14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.b","version":"48'10","reported_seq":137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.796083+0000","last_change":"2024-04-26T01:24:24.952945+0000","last_active":"2024-04-26T01:24:26.796083+0000","last_peered":"2024-04-26T01:24:26.796083+0000","last_clean":"2024-04-26T01:24:26.796083+0000","last_became_active":"2024-04-26T01:24:05.273930+0000","last_became_peered":"2024-04-26T01:24:05.273930+0000","last_unstale":"2024-04-26T01:24:26.796083+0000","last_undegraded":"2024-04-26T01:24:26.796083+0000","last_fullsized":"2024-04-26T01:24:26.796083+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.245183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10120567499999999,"stat_sum":{"num_bytes":81920,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":573,"num_write":23,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,9,6,3,13],"acting":[4,0,1,9,6,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"3.a","version":"44'15","reported_seq":149,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-26T01:24:31.088741+0000","last_change":"2024-04-26T01:24:31.089971+0000","last_active":"2024-04-26T01:24:31.088741+0000","last_peered":"2024-04-26T01:24:31.088741+0000","last_clean":"2024-04-26T01:24:31.088741+0000","last_became_active":"2024-04-26T01:24:01.227642+0000","last_became_peered":"2024-04-26T01:24:01.227642+0000","last_unstale":"2024-04-26T01:24:31.088741+0000","last_undegraded":"2024-04-26T01:24:31.088741+0000","last_fullsized":"2024-04-26T01:24:31.088741+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.139748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00151727,"stat_sum":{"num_bytes":65536,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":25,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,1,8,14,6,11],"acting":[3,9,1,8,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9","length":"5"}]},{"pgid":"3.9","version":"53'22","reported_seq":141,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.817622+0000","last_change":"2024-04-26T01:24:25.154830+0000","last_active":"2024-04-26T01:24:26.817622+0000","last_peered":"2024-04-26T01:24:26.817622+0000","last_clean":"2024-04-26T01:24:26.817622+0000","last_became_active":"2024-04-26T01:24:01.291346+0000","last_became_peered":"2024-04-26T01:24:01.291346+0000","last_unstale":"2024-04-26T01:24:26.817622+0000","last_undegraded":"2024-04-26T01:24:26.817622+0000","last_fullsized":"2024-04-26T01:24:26.817622+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.912177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27000373100000002,"stat_sum":{"num_bytes":1540096,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":17,"num_read_kb":1013,"num_write":12,"num_write_kb":1536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"3.8","version":"59'57","reported_seq":237,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T01:24:32.081208+0000","last_change":"2024-04-26T01:24:31.137363+0000","last_active":"2024-04-26T01:24:32.081208+0000","last_peered":"2024-04-26T01:24:32.081208+0000","last_clean":"2024-04-26T01:24:32.081208+0000","last_became_active":"2024-04-26T01:24:16.447854+0000","last_became_peered":"2024-04-26T01:24:16.447854+0000","last_unstale":"2024-04-26T01:24:32.081208+0000","last_undegraded":"2024-04-26T01:24:32.081208+0000","last_fullsized":"2024-04-26T01:24:32.081208+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:24.524246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.214096435,"stat_sum":{"num_bytes":7979008,"num_objects":12,"num_object_clones":7,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":33,"num_read_kb":11229,"num_write":51,"num_write_kb":8992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4161536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1,9,14,15,6],"acting":[5,0,1,9,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"9","length":"5"}]},{"pgid":"1.0","version":"34'157","reported_seq":281,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.028041+0000","last_change":"2024-04-26T01:24:19.157462+0000","last_active":"2024-04-26T01:24:28.028041+0000","last_peered":"2024-04-26T01:24:28.028041+0000","last_clean":"2024-04-26T01:24:28.028041+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:24:28.028041+0000","last_undegraded":"2024-04-26T01:24:28.028041+0000","last_fullsized":"2024-04-26T01:24:28.028041+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:23:48.870422+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:54.023472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-26T01:24:31.881036+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:24:31.881036+0000","last_peered":"2024-04-26T01:24:31.881036+0000","last_clean":"2024-04-26T01:24:31.881036+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:24:31.881036+0000","last_undegraded":"2024-04-26T01:24:31.881036+0000","last_fullsized":"2024-04-26T01:24:31.881036+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"56'31","reported_seq":171,"reported_epoch":56,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T01:24:28.133596+0000","last_change":"2024-04-26T01:24:27.929318+0000","last_active":"2024-04-26T01:24:28.133596+0000","last_peered":"2024-04-26T01:24:28.133596+0000","last_clean":"2024-04-26T01:24:28.133596+0000","last_became_active":"2024-04-26T01:24:16.348216+0000","last_became_peered":"2024-04-26T01:24:16.348216+0000","last_unstale":"2024-04-26T01:24:28.133596+0000","last_undegraded":"2024-04-26T01:24:28.133596+0000","last_fullsized":"2024-04-26T01:24:28.133596+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.592546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20613052000000001,"stat_sum":{"num_bytes":3178496,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":342,"num_write":17,"num_write_kb":3824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-26T01:24:30.989673+0000","last_change":"2024-04-26T01:23:52.211699+0000","last_active":"2024-04-26T01:24:30.989673+0000","last_peered":"2024-04-26T01:24:30.989673+0000","last_clean":"2024-04-26T01:24:30.989673+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:24:30.989673+0000","last_undegraded":"2024-04-26T01:24:30.989673+0000","last_fullsized":"2024-04-26T01:24:30.989673+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:51.317271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"45'14","reported_seq":149,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.896080+0000","last_change":"2024-04-26T01:24:25.053594+0000","last_active":"2024-04-26T01:24:26.896080+0000","last_peered":"2024-04-26T01:24:26.896080+0000","last_clean":"2024-04-26T01:24:26.896080+0000","last_became_active":"2024-04-26T01:24:16.442216+0000","last_became_peered":"2024-04-26T01:24:16.442216+0000","last_unstale":"2024-04-26T01:24:26.896080+0000","last_undegraded":"2024-04-26T01:24:26.896080+0000","last_fullsized":"2024-04-26T01:24:26.896080+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.617908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101877333,"stat_sum":{"num_bytes":2621440,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":4406,"num_write":10,"num_write_kb":3312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,14,13],"acting":[4,11,9,5,10,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":69,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.928015+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:24:27.928015+0000","last_peered":"2024-04-26T01:24:27.928015+0000","last_clean":"2024-04-26T01:24:27.928015+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:24:27.928015+0000","last_undegraded":"2024-04-26T01:24:27.928015+0000","last_fullsized":"2024-04-26T01:24:27.928015+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"55'34","reported_seq":186,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.228083+0000","last_change":"2024-04-26T01:24:24.844011+0000","last_active":"2024-04-26T01:24:27.228083+0000","last_peered":"2024-04-26T01:24:27.228083+0000","last_clean":"2024-04-26T01:24:27.228083+0000","last_became_active":"2024-04-26T01:24:16.549243+0000","last_became_peered":"2024-04-26T01:24:16.549243+0000","last_unstale":"2024-04-26T01:24:27.228083+0000","last_undegraded":"2024-04-26T01:24:27.228083+0000","last_fullsized":"2024-04-26T01:24:27.228083+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:24.413010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017168909999999999,"stat_sum":{"num_bytes":6176768,"num_objects":12,"num_object_clones":4,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":31,"num_read_kb":5342,"num_write":28,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-26T01:24:31.188590+0000","last_change":"2024-04-26T01:24:05.176463+0000","last_active":"2024-04-26T01:24:31.188590+0000","last_peered":"2024-04-26T01:24:31.188590+0000","last_clean":"2024-04-26T01:24:31.188590+0000","last_became_active":"2024-04-26T01:24:05.175683+0000","last_became_peered":"2024-04-26T01:24:05.175683+0000","last_unstale":"2024-04-26T01:24:31.188590+0000","last_undegraded":"2024-04-26T01:24:31.188590+0000","last_fullsized":"2024-04-26T01:24:31.188590+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:57.025465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"35'2","reported_seq":140,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.029043+0000","last_change":"2024-04-26T01:24:28.029612+0000","last_active":"2024-04-26T01:24:28.029043+0000","last_peered":"2024-04-26T01:24:28.029043+0000","last_clean":"2024-04-26T01:24:28.029043+0000","last_became_active":"2024-04-26T01:24:16.644086+0000","last_became_peered":"2024-04-26T01:24:16.644086+0000","last_unstale":"2024-04-26T01:24:28.029043+0000","last_undegraded":"2024-04-26T01:24:28.029043+0000","last_fullsized":"2024-04-26T01:24:28.029043+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:14.994473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091882499999999996,"stat_sum":{"num_bytes":622592,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,4,15,6,5,3],"acting":[13,9,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"3"}]},{"pgid":"3.5","version":"0'0","reported_seq":128,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.823583+0000","last_change":"2024-04-26T01:24:24.884359+0000","last_active":"2024-04-26T01:24:26.823583+0000","last_peered":"2024-04-26T01:24:26.823583+0000","last_clean":"2024-04-26T01:24:26.823583+0000","last_became_active":"2024-04-26T01:24:16.447258+0000","last_became_peered":"2024-04-26T01:24:16.447258+0000","last_unstale":"2024-04-26T01:24:26.823583+0000","last_undegraded":"2024-04-26T01:24:26.823583+0000","last_fullsized":"2024-04-26T01:24:26.823583+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:00.481418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001182701,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,1,0,6],"acting":[15,8,4,13,1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":66,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.795950+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:24:26.795950+0000","last_peered":"2024-04-26T01:24:26.795950+0000","last_clean":"2024-04-26T01:24:26.795950+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:24:26.795950+0000","last_undegraded":"2024-04-26T01:24:26.795950+0000","last_fullsized":"2024-04-26T01:24:26.795950+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"41'12","reported_seq":166,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.776588+0000","last_change":"2024-04-26T01:24:28.125764+0000","last_active":"2024-04-26T01:24:28.776588+0000","last_peered":"2024-04-26T01:24:28.776588+0000","last_clean":"2024-04-26T01:24:28.776588+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:24:28.776588+0000","last_undegraded":"2024-04-26T01:24:28.776588+0000","last_fullsized":"2024-04-26T01:24:28.776588+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:21.162880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011626659999999999,"stat_sum":{"num_bytes":1490944,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":6103,"num_write":8,"num_write_kb":1456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"9","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.923352+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:24:26.923352+0000","last_peered":"2024-04-26T01:24:26.923352+0000","last_clean":"2024-04-26T01:24:26.923352+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:24:26.923352+0000","last_undegraded":"2024-04-26T01:24:26.923352+0000","last_fullsized":"2024-04-26T01:24:26.923352+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"55'35","reported_seq":164,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:27.336662+0000","last_change":"2024-04-26T01:24:24.852327+0000","last_active":"2024-04-26T01:24:27.336662+0000","last_peered":"2024-04-26T01:24:27.336662+0000","last_clean":"2024-04-26T01:24:27.336662+0000","last_became_active":"2024-04-26T01:24:01.091412+0000","last_became_peered":"2024-04-26T01:24:01.091412+0000","last_unstale":"2024-04-26T01:24:27.336662+0000","last_undegraded":"2024-04-26T01:24:27.336662+0000","last_fullsized":"2024-04-26T01:24:27.336662+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.458801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089147,"stat_sum":{"num_bytes":3391488,"num_objects":8,"num_object_clones":3,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":32,"num_read_kb":5195,"num_write":19,"num_write_kb":4224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-26T01:24:28.775393+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:24:28.775393+0000","last_peered":"2024-04-26T01:24:28.775393+0000","last_clean":"2024-04-26T01:24:28.775393+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:24:28.775393+0000","last_undegraded":"2024-04-26T01:24:28.775393+0000","last_fullsized":"2024-04-26T01:24:28.775393+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:10.471688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"55'25","reported_seq":160,"reported_epoch":56,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T01:24:27.804697+0000","last_change":"2024-04-26T01:24:27.804737+0000","last_active":"2024-04-26T01:24:27.804697+0000","last_peered":"2024-04-26T01:24:27.804697+0000","last_clean":"2024-04-26T01:24:27.804697+0000","last_became_active":"2024-04-26T01:24:05.252337+0000","last_became_peered":"2024-04-26T01:24:05.252337+0000","last_unstale":"2024-04-26T01:24:27.804697+0000","last_undegraded":"2024-04-26T01:24:27.804697+0000","last_fullsized":"2024-04-26T01:24:27.804697+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:07.504657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20108716700000001,"stat_sum":{"num_bytes":2834432,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":17,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":49152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,11,6,9],"acting":[0,13,1,15,11,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"1"},{"start":"5","length":"3"},{"start":"9","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":67,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-26T01:24:26.823363+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:24:26.823363+0000","last_peered":"2024-04-26T01:24:26.823363+0000","last_clean":"2024-04-26T01:24:26.823363+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:24:26.823363+0000","last_undegraded":"2024-04-26T01:24:26.823363+0000","last_fullsized":"2024-04-26T01:24:26.823363+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":42057728,"num_objects":86,"num_object_clones":36,"num_object_copies":602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":5,"num_read":263,"num_read_kb":45693,"num_write":292,"num_write_kb":49120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21413888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77717504,"data_stored":100634624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366,"ondisk_log_size":366,"up":112,"acting":112,"num_store_stats":15},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":0,"seq":94489280524,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":5124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":5246976,"data_stored":6049050,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280523,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37936,"kb_used_data":6756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333904,"statfs":{"total":96636764160,"available":96597917696,"internally_reserved":0,"allocated":6918144,"data_stored":9256218,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34196,"kb_used_data":7176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337644,"statfs":{"total":96636764160,"available":96601747456,"internally_reserved":0,"allocated":7348224,"data_stored":9162010,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313227,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35148,"kb_used_data":8128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336692,"statfs":{"total":96636764160,"available":96600772608,"internally_reserved":0,"allocated":8323072,"data_stored":10308890,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313226,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36936,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334904,"statfs":{"total":96636764160,"available":96598941696,"internally_reserved":0,"allocated":5890048,"data_stored":7720218,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,4,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313228,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34112,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337728,"statfs":{"total":96636764160,"available":96601833472,"internally_reserved":0,"allocated":2994176,"data_stored":3887246,"data_compressed":143420,"data_compressed_allocated":192512,"data_compressed_original":827392,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313226,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":5000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":5120000,"data_stored":5491994,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313228,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":6888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":7053312,"data_stored":9125486,"data_compressed":109365,"data_compressed_allocated":151552,"data_compressed_original":303104,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313227,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":9004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":9220096,"data_stored":12164378,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":90194313227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313227,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":5904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":6045696,"data_stored":7519514,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313226,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":4592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":4702208,"data_stored":5864730,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313228,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33492,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338348,"statfs":{"total":96636764160,"available":96602468352,"internally_reserved":0,"allocated":6627328,"data_stored":8797466,"data_compressed":97326,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313227,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39692,"kb_used_data":4344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332148,"statfs":{"total":96636764160,"available":96596119552,"internally_reserved":0,"allocated":4448256,"data_stored":5799738,"data_compressed":129943,"data_compressed_allocated":176128,"data_compressed_original":782336,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6311936,"data_stored":8568832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6742016,"data_stored":8761344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2613248,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4513792,"data_stored":5091328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6647808,"data_stored":9408512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8663040,"data_stored":11726848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4640768,"data_stored":5648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5488640,"data_stored":7081984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096000,"data_stored":5464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6668288,"data_stored":8818688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":3809280,"data_stored":4898816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":7737344,"data_stored":9928704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5332992,"data_stored":7368704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:24:33.556 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-26T01:24:33.556 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:24:33.557 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-04-26T01:24:33.557 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-04-26T01:24:33.599 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:33.603+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:33.624 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:33.624 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:33.624 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:33.624 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:33.640 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:33.696 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:33.695+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:33.698 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing write tid 5 to smithi18128752-45 2024-04-26T01:24:33.699 INFO:tasks.rados.rados.0.smithi181.stdout:189: oid 45 updating version 0 to 53 2024-04-26T01:24:33.699 INFO:tasks.rados.rados.0.smithi181.stdout:189: oid 45 updating version 53 to 56 2024-04-26T01:24:33.699 INFO:tasks.rados.rados.0.smithi181.stdout:189: oid 45 version 56 is already newer than 55 2024-04-26T01:24:33.699 INFO:tasks.rados.rados.0.smithi181.stdout:189: oid 45 version 56 is already newer than 54 2024-04-26T01:24:33.699 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 56 (ObjNum 80 snap 13 seq_num 80) dirty exists 2024-04-26T01:24:33.699 INFO:tasks.rados.rados.0.smithi181.stdout:189: left oid 45 (ObjNum 80 snap 13 seq_num 80) 2024-04-26T01:24:33.762 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:33.762 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:33.762 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:33.762 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:33.762 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:33.767 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-04-26T01:24:33.773 INFO:tasks.rados.rados.0.smithi181.stdout:186: done (6 left) 2024-04-26T01:24:33.774 INFO:tasks.rados.rados.0.smithi181.stdout:189: done (5 left) 2024-04-26T01:24:33.774 INFO:tasks.rados.rados.0.smithi181.stdout:191: done (4 left) 2024-04-26T01:24:33.774 INFO:tasks.rados.rados.0.smithi181.stdout:192: done (3 left) 2024-04-26T01:24:33.774 INFO:tasks.rados.rados.0.smithi181.stdout:194: done (2 left) 2024-04-26T01:24:33.774 INFO:tasks.rados.rados.0.smithi181.stdout:195: snap_create 2024-04-26T01:24:33.776 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:33.797 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:33.795+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:33.799 INFO:tasks.rados.rados.0.smithi181.stdout:193: finishing write tid 5 to smithi18128752-9 2024-04-26T01:24:33.799 INFO:tasks.rados.rados.0.smithi181.stdout:193: oid 9 updating version 0 to 43 2024-04-26T01:24:33.799 INFO:tasks.rados.rados.0.smithi181.stdout:193: oid 9 updating version 43 to 44 2024-04-26T01:24:33.799 INFO:tasks.rados.rados.0.smithi181.stdout:193: oid 9 updating version 44 to 45 2024-04-26T01:24:33.799 INFO:tasks.rados.rados.0.smithi181.stdout:193: oid 9 version 45 is already newer than 42 2024-04-26T01:24:33.799 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 45 (ObjNum 82 snap 13 seq_num 82) dirty exists 2024-04-26T01:24:33.799 INFO:tasks.rados.rados.0.smithi181.stdout:193: left oid 9 (ObjNum 82 snap 13 seq_num 82) 2024-04-26T01:24:33.800 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 5 to smithi18128752-20 2024-04-26T01:24:33.801 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 20 updating version 0 to 12 2024-04-26T01:24:33.801 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 20 updating version 12 to 14 2024-04-26T01:24:33.801 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 20 version 14 is already newer than 13 2024-04-26T01:24:33.801 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 20 updating version 14 to 15 2024-04-26T01:24:33.801 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 15 (ObjNum 81 snap 13 seq_num 81) dirty exists 2024-04-26T01:24:33.801 INFO:tasks.rados.rados.0.smithi181.stdout:190: left oid 20 (ObjNum 81 snap 13 seq_num 81) 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 35/34 ECSubWrite(tid=5, reqid=client.4589.0:116, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:33.898 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.341623642999998, 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20106410399999999, 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.899 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559045+0000", 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8929999999999999e-06 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.900 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559097+0000", 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2672e-05 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559128+0000", 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1019000000000002e-05 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659233+0000", 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100104779 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.901 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659393+0000", 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016010199999999999 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659420+0000", 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6333e-05 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760047+0000", 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100627094 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.902 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760107+0000", 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0211999999999999e-05 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s3 35/30 ECSubWrite(tid=167, reqid=client.4589.0:131, at_version=35'6, trim_to=0'0, roll_forward_to=33'4))", 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.659832+0000", 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.240834153000002, 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10299027099999999, 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.903 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659832+0000", 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659832+0000", 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.904 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659833+0000", 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.4700000000000001e-07 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659877+0000", 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3801999999999998e-05 2024-04-26T01:24:33.905 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.906 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.906 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659897+0000", 2024-04-26T01:24:33.906 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0625999999999999e-05 2024-04-26T01:24:33.906 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.906 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659916+0000", 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8510000000000001e-05 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.761482+0000", 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101566559 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.761497+0000", 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4676e-05 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.907 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.762751+0000", 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012543330000000001 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.762823+0000", 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.1118000000000006e-05 2024-04-26T01:24:33.908 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s3 36/30 ECSubWrite(tid=172, reqid=client.4589.0:167, at_version=36'7, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.407368+0000", 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.493298404000001, 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104738489, 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.909 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407368+0000", 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407368+0000", 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407369+0000", 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.237e-06 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.910 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407683+0000", 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031392099999999999 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.407712+0000", 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8493e-05 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.507844+0000", 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100132422 2024-04-26T01:24:33.911 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.508011+0000", 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000167355 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.508041+0000", 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9280000000000001e-05 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.912 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.512030+0000", 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0039888359999999999 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.512106+0000", 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6945000000000003e-05 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.es5 40/34 ECSubWrite(tid=40, reqid=client.4589.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=40'23))", 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:33.913 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 21.891192428, 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30174757699999999, 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:33.914 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009476+0000", 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9800000000000001e-06 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009502+0000", 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6495999999999998e-05 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009541+0000", 2024-04-26T01:24:33.915 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8748999999999998e-05 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009564+0000", 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2773e-05 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109788+0000", 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100224176 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.916 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109803+0000", 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4487e-05 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311086+0000", 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201283252 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311222+0000", 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013566399999999999 2024-04-26T01:24:33.917 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 41/34 ECSubWrite(tid=27, reqid=osd.0.0:27, at_version=41'17, trim_to=0'0, roll_forward_to=39'15))", 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 21.801218898999998, 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21182642600000001, 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.918 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7300000000000001e-07 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.919 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099485+0000", 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7493000000000002e-05 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099509+0000", 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.387e-05 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099527+0000", 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7598999999999999e-05 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.920 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110237+0000", 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.010710366000000001 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110253+0000", 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5319999999999999e-05 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311234+0000", 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20098134600000001 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.921 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311274+0000", 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9959000000000001e-05 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=32, reqid=client.4589.0:266, at_version=41'17, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 21.725520367000001, 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13212590199999999, 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.922 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.923 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175147+0000", 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.13e-07 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175731+0000", 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000583973 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175761+0000", 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0536e-05 2024-04-26T01:24:33.924 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175777+0000", 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6076e-05 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302772+0000", 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12699442499999999 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.925 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302793+0000", 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1382999999999998e-05 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307218+0000", 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0044247139999999997 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307272+0000", 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4181999999999997e-05 2024-04-26T01:24:33.926 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=33, reqid=osd.5.0:33, at_version=41'19, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 21.722547984999999, 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.129420758, 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.927 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178120+0000", 2024-04-26T01:24:33.928 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178160+0000", 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9356000000000001e-05 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178190+0000", 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0179e-05 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178206+0000", 2024-04-26T01:24:33.929 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5809000000000001e-05 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.304772+0000", 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12656614899999999 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.304787+0000", 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4893e-05 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.930 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307500+0000", 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027131490000000002 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307539+0000", 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9486000000000001e-05 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.931 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=34, reqid=client.4589.0:267, at_version=41'20, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.181203+0000", 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 21.719463644000001, 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.127937575, 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181203+0000", 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.932 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181203+0000", 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181205+0000", 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181296+0000", 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1978000000000006e-05 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.933 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.181326+0000", 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9998999999999999e-05 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.281469+0000", 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10014216500000001 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.306125+0000", 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.024655928000000001 2024-04-26T01:24:33.934 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.306138+0000", 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3003000000000001e-05 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309045+0000", 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029073609999999998 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309140+0000", 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5397999999999997e-05 2024-04-26T01:24:33.935 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs4 45/34 ECSubWrite(tid=42, reqid=client.4589.0:297, at_version=45'21, trim_to=0'0, roll_forward_to=45'19))", 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:16.751305+0000", 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.149361020000001, 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10363475599999999, 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.936 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751305+0000", 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751305+0000", 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751307+0000", 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.593e-06 2024-04-26T01:24:33.937 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751714+0000", 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00040656500000000001 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751741+0000", 2024-04-26T01:24:33.938 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7239000000000001e-05 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.751761+0000", 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0486e-05 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.852002+0000", 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100241011 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.852034+0000", 2024-04-26T01:24:33.939 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1655000000000001e-05 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.854873+0000", 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028389719999999999 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:16.854940+0000", 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7235000000000003e-05 2024-04-26T01:24:33.940 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=68, reqid=client.4589.0:313, at_version=46'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.936766857, 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20854388500000001, 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.941 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.942 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963902+0000", 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964264+0000", 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036215900000000003 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.943 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964306+0000", 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1801999999999997e-05 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964334+0000", 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8133e-05 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968462+0000", 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041272710000000001 2024-04-26T01:24:33.944 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968486+0000", 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4219999999999999e-05 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172363+0000", 2024-04-26T01:24:33.945 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20387691099999999 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172443+0000", 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0798999999999997e-05 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=69, reqid=client.4589.0:314, at_version=46'19, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:33.946 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.932603917, 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20440952700000001, 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.947 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:33.949 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.949 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.949 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.949 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.949 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968064+0000", 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9829999999999999e-06 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968174+0000", 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000109028 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968220+0000", 2024-04-26T01:24:33.950 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6106000000000002e-05 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968244+0000", 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4239e-05 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970652+0000", 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024076810000000001 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970665+0000", 2024-04-26T01:24:33.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3652000000000001e-05 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172450+0000", 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201785134 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172472+0000", 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1704000000000001e-05 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=70, reqid=client.4589.0:315, at_version=46'20, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.92972576, 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201814353, 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970942+0000", 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8339999999999999e-06 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971003+0000", 2024-04-26T01:24:33.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0401e-05 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971040+0000", 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7469999999999999e-05 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971064+0000", 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3609e-05 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.171968+0000", 2024-04-26T01:24:33.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200904202 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.171991+0000", 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2702e-05 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172718+0000", 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00072753100000000001 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172755+0000", 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6603999999999997e-05 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.7s1 47/30 ECSubWrite(tid=126, reqid=client.4589.0:327, at_version=47'30, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:18.558729+0000", 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.341937593000001, 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.108543682, 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.957 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.558729+0000", 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.558729+0000", 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.558729+0000", 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8200000000000001e-07 2024-04-26T01:24:33.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.558747+0000", 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7768999999999999e-05 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.558753+0000", 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1260000000000003e-06 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.558758+0000", 2024-04-26T01:24:33.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5399999999999997e-06 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.665219+0000", 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.106461028 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.665250+0000", 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1176000000000001e-05 2024-04-26T01:24:33.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.667154+0000", 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001904134 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.667272+0000", 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000118627 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 48/44 ECSubWrite(tid=55, reqid=client.4589.0:335, at_version=48'31, trim_to=0'0, roll_forward_to=47'27))", 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:19.287391+0000", 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 14.613275033000001, 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103228262, 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287391+0000", 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287391+0000", 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287392+0000", 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.77e-07 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287777+0000", 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00038541600000000002 2024-04-26T01:24:33.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287805+0000", 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8293000000000002e-05 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287826+0000", 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0392999999999999e-05 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287934+0000", 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010796100000000001 2024-04-26T01:24:33.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.287945+0000", 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0891e-05 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.390557+0000", 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102612299 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.390620+0000", 2024-04-26T01:24:33.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2632000000000006e-05 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 51/44 ECSubWrite(tid=67, reqid=client.4589.0:382, at_version=51'41, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.215479+0000", 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.685186972, 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10504985999999999, 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.215479+0000", 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.215479+0000", 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.215480+0000", 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6300000000000005e-07 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.216066+0000", 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00058610599999999998 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.216101+0000", 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4588999999999997e-05 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.216125+0000", 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3896e-05 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.217965+0000", 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001840789 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.217993+0000", 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7926e-05 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.320436+0000", 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102442372 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.320529+0000", 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3518999999999997e-05 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 56/44 ECSubWrite(tid=107, reqid=osd.14.0:107, at_version=56'36, trim_to=0'0, roll_forward_to=55'33))", 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 5.7511298929999999, 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20195263099999999, 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.971 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149538+0000", 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149588+0000", 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0337000000000003e-05 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149623+0000", 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4782e-05 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149641+0000", 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7886000000000001e-05 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.349947+0000", 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20030594800000001 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.349962+0000", 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5746000000000001e-05 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.351420+0000", 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014574519999999999 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.351489+0000", 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9412999999999994e-05 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.bs4 61/34 ECSubWrite(tid=175, reqid=client.4589.0:473, at_version=61'12, trim_to=0'0, roll_forward_to=48'10))", 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.884288+0000", 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.0163783930000001, 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104728393, 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.975 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.884288+0000", 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.884288+0000", 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.976 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.884289+0000", 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.884859+0000", 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057025200000000004 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.884889+0000", 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0412999999999998e-05 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.977 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.884912+0000", 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2432000000000001e-05 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.985248+0000", 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100336568 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.985266+0000", 2024-04-26T01:24:33.978 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7479e-05 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.988959+0000", 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036926839999999999 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.989016+0000", 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7895e-05 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.979 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.7s1 61/30 ECSubWrite(tid=177, reqid=client.4589.0:478, at_version=61'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.900325+0000", 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.0003411090000001, 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10725473200000001, 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.980 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.900325+0000", 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.900325+0000", 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.981 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.900327+0000", 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6869999999999999e-06 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.900462+0000", 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013469299999999999 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.900493+0000", 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1272000000000001e-05 2024-04-26T01:24:33.982 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.000599+0000", 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100106261 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.000774+0000", 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017447099999999999 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.983 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.000789+0000", 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5500000000000001e-05 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.007468+0000", 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0066788109999999998 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.984 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.007580+0000", 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000112037 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 61/44 ECSubWrite(tid=127, reqid=client.4589.0:471, at_version=61'56, trim_to=0'0, roll_forward_to=59'47))", 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.972287+0000", 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.92837905300000001, 2024-04-26T01:24:33.985 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17427974299999999, 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972287+0000", 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972287+0000", 2024-04-26T01:24:33.986 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972290+0000", 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.447e-06 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972359+0000", 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9303999999999999e-05 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.987 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972396+0000", 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6665999999999998e-05 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972420+0000", 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4392e-05 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.145794+0000", 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.173373588 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.145802+0000", 2024-04-26T01:24:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6510000000000007e-06 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.146516+0000", 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00071390400000000001 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.146567+0000", 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0791000000000002e-05 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.989 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds6 61/44 ECSubWrite(tid=182, reqid=osd.4.0:178, at_version=0'0, trim_to=0'0, roll_forward_to=61'33))", 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.49825050700000001, 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201923306, 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:33.990 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.991 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402417+0000", 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1299999999999998e-07 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402437+0000", 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0322e-05 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402460+0000", 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2972999999999998e-05 2024-04-26T01:24:33.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402512+0000", 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2151999999999998e-05 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.603076+0000", 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20056403 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.603093+0000", 2024-04-26T01:24:33.993 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6336000000000001e-05 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.604291+0000", 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001198253 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.604339+0000", 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8327e-05 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.994 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.995 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:33.995 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:33.995 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:33.995 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:33.899+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:33.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:33.997 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T01:24:33.923+0000 7f533fe62640 -1 mon.a@0(leader).osd e62 definitely_dead 0 2024-04-26T01:24:34.041 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:34.043+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:34.100 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:34.099+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:34.169 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:34.169 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:34.169 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:34.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:34.228 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:34.231+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:34.302 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:34.303+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:34.308 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:34.308 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:34.308 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:34.308 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:34.308 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:34.322 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:34.412 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:34.415+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/30 ECSubWrite(tid=42, reqid=client.4589.0:122, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.087029+0000", 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.407492016999999, 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101532873, 2024-04-26T01:24:34.492 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087029+0000", 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087029+0000", 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.493 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087031+0000", 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9130000000000001e-06 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087054+0000", 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2688999999999999e-05 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.494 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087083+0000", 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8856999999999998e-05 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.087098+0000", 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5628999999999999e-05 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.495 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.187328+0000", 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100230262 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.187344+0000", 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5458999999999999e-05 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.496 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188505+0000", 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011609109999999999 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188562+0000", 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7153e-05 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/30 ECSubWrite(tid=44, reqid=client.4589.0:137, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:34.497 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.188045+0000", 2024-04-26T01:24:34.498 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.306476537000002, 2024-04-26T01:24:34.498 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10172057599999999, 2024-04-26T01:24:34.498 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.498 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.498 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.498 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.498 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188045+0000", 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188045+0000", 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.499 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188047+0000", 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1600000000000001e-06 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188078+0000", 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0852999999999998e-05 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.500 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188110+0000", 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2026000000000001e-05 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.188152+0000", 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1919000000000003e-05 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.288563+0000", 2024-04-26T01:24:34.501 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10041122299999999 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.288593+0000", 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0369e-05 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.289697+0000", 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001103971 2024-04-26T01:24:34.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.289765+0000", 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8054999999999998e-05 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/30 ECSubWrite(tid=47, reqid=client.4589.0:140, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:34.503 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.201602212000001, 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20188384300000001, 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.504 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292921+0000", 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8870000000000001e-06 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292964+0000", 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3152000000000003e-05 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.505 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.293004+0000", 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0065999999999999e-05 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.393103+0000", 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100099122 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493412+0000", 2024-04-26T01:24:34.506 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10030921299999999 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493428+0000", 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494705+0000", 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012777590000000001 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.507 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494803+0000", 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7477000000000002e-05 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/30 ECSubWrite(tid=39, reqid=client.4589.0:154, at_version=35'7, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:34.508 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.662714+0000", 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.831807086000001, 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10209053999999999, 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662714+0000", 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.509 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662714+0000", 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662716+0000", 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.187e-06 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.510 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662795+0000", 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9223999999999999e-05 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662827+0000", 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1678999999999999e-05 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662843+0000", 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6223e-05 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.763113+0000", 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100269335 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.763132+0000", 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.893e-05 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764708+0000", 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001576647 2024-04-26T01:24:34.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764805+0000", 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6315000000000002e-05 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/34 ECSubWrite(tid=18, reqid=client.4589.0:77, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.764464+0000", 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.730057340999998, 2024-04-26T01:24:34.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101881215, 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764464+0000", 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764464+0000", 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764466+0000", 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8139999999999999e-06 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764490+0000", 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4304999999999999e-05 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764517+0000", 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6869999999999999e-05 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764541+0000", 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4629000000000001e-05 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764694+0000", 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015242599999999999 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.764718+0000", 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4026e-05 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.866267+0000", 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101548914 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.866345+0000", 2024-04-26T01:24:34.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8231000000000004e-05 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 36/34 ECSubWrite(tid=26, reqid=client.4589.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.814226735999998, 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200889605, 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.518 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680296+0000", 2024-04-26T01:24:34.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8640000000000001e-06 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680320+0000", 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4202e-05 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680350+0000", 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9998999999999999e-05 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780494+0000", 2024-04-26T01:24:34.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100143549 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780762+0000", 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026773600000000003 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780776+0000", 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881147+0000", 2024-04-26T01:24:34.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100370127 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881184+0000", 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7422000000000002e-05 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 37/30 ECSubWrite(tid=47, reqid=client.4589.0:198, at_version=37'11, trim_to=0'0, roll_forward_to=35'7))", 2024-04-26T01:24:34.522 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.115662188999998, 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10198423299999999, 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378859+0000", 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8299999999999997e-07 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378883+0000", 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3235999999999999e-05 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378894+0000", 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1579999999999999e-05 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.378903+0000", 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.2059999999999996e-06 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.479752+0000", 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100848803 2024-04-26T01:24:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.479771+0000", 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8383e-05 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480783+0000", 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010124540000000001 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.480843+0000", 2024-04-26T01:24:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0087999999999998e-05 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/30 ECSubWrite(tid=90, reqid=osd.4.0:90, at_version=41'10, trim_to=0'0, roll_forward_to=37'8))", 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.294868035, 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103318356, 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199654+0000", 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199704+0000", 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9656e-05 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199735+0000", 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1396000000000002e-05 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199762+0000", 2024-04-26T01:24:34.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7092000000000001e-05 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.299984+0000", 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100221583 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.300000+0000", 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5906999999999999e-05 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302870+0000", 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028700879999999998 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302971+0000", 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010186799999999999 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.531 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 41/34 ECSubWrite(tid=33, reqid=osd.11.0:33, at_version=41'13, trim_to=0'0, roll_forward_to=37'11))", 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.207265+0000", 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.287255883, 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101853824, 2024-04-26T01:24:34.532 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207265+0000", 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207265+0000", 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207266+0000", 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4299999999999999e-07 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207334+0000", 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8591999999999997e-05 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207357+0000", 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2943e-05 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.207373+0000", 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5486e-05 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307636+0000", 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100263175 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307654+0000", 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8627e-05 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309043+0000", 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001388303 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.309119+0000", 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6354999999999997e-05 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.537 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.092478906, 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201895835, 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402045+0000", 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2800000000000002e-06 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402074+0000", 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9366e-05 2024-04-26T01:24:34.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402148+0000", 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4255e-05 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502254+0000", 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100106322 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502424+0000", 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000169259 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502449+0000", 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4909000000000002e-05 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603856+0000", 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101407649 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603938+0000", 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1794999999999999e-05 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/30 ECSubWrite(tid=92, reqid=osd.4.0:90, at_version=0'0, trim_to=0'0, roll_forward_to=41'10))", 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.502715+0000", 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 21.991805683999999, 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10209800300000001, 2024-04-26T01:24:34.542 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502715+0000", 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502715+0000", 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502717+0000", 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.606e-06 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502742+0000", 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5409e-05 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502770+0000", 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7540000000000001e-05 2024-04-26T01:24:34.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502793+0000", 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2739e-05 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603415+0000", 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100622239 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603444+0000", 2024-04-26T01:24:34.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9224999999999999e-05 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.604744+0000", 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0013000659999999999 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.604813+0000", 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9178999999999996e-05 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 45/34 ECSubWrite(tid=43, reqid=client.4589.0:297, at_version=0'0, trim_to=0'0, roll_forward_to=45'21))", 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.048297+0000", 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.446224537999999, 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101580799, 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048297+0000", 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048297+0000", 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048299+0000", 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.04e-06 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048323+0000", 2024-04-26T01:24:34.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4413000000000002e-05 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048355+0000", 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1619000000000003e-05 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.048372+0000", 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.732e-05 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.148664+0000", 2024-04-26T01:24:34.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100291619 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.148680+0000", 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6475999999999999e-05 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.149792+0000", 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001111807 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.149877+0000", 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5505000000000002e-05 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 48/34 ECSubWrite(tid=49, reqid=client.4589.0:334, at_version=48'23, trim_to=0'0, roll_forward_to=45'21))", 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:24:34.551 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.129671904, 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103332259, 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.552 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364850+0000", 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0299999999999998e-07 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365180+0000", 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032962699999999998 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365243+0000", 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3516000000000006e-05 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365268+0000", 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4522e-05 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365458+0000", 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000189946 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365484+0000", 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6800000000000001e-05 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.468084+0000", 2024-04-26T01:24:34.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102600071 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.468182+0000", 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7073999999999999e-05 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 48/34 ECSubWrite(tid=50, reqid=client.4589.0:334, at_version=0'0, trim_to=0'0, roll_forward_to=48'23))", 2024-04-26T01:24:34.556 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:19.570603+0000", 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 14.923917969, 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10192712800000001, 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.570603+0000", 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.570603+0000", 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.570605+0000", 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.694e-06 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.570631+0000", 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5643000000000002e-05 2024-04-26T01:24:34.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.570664+0000", 2024-04-26T01:24:34.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3175e-05 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.670765+0000", 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100101154 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.670943+0000", 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017807000000000001 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.670977+0000", 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3976000000000001e-05 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.672434+0000", 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001456988 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.672530+0000", 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6428000000000006e-05 2024-04-26T01:24:34.563 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 51/44 ECSubWrite(tid=206, reqid=client.4589.0:388, at_version=51'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.151577152, 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.138940597, 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.564 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342946+0000", 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343263+0000", 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000317664 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343300+0000", 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6293e-05 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343321+0000", 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1605999999999999e-05 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.479650+0000", 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.136328692 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.479665+0000", 2024-04-26T01:24:34.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5393999999999999e-05 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.481807+0000", 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021412390000000001 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.481885+0000", 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7959000000000004e-05 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 51/34 ECSubWrite(tid=59, reqid=client.4589.0:397, at_version=51'30, trim_to=0'0, roll_forward_to=49'28))", 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.940932794, 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10403118, 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.569 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553589+0000", 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6700000000000003e-07 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553663+0000", 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3734999999999998e-05 2024-04-26T01:24:34.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553697+0000", 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4003e-05 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553722+0000", 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4678999999999999e-05 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.653970+0000", 2024-04-26T01:24:34.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100247954 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.653997+0000", 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7089000000000001e-05 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.657525+0000", 2024-04-26T01:24:34.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0035286039999999999 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.657620+0000", 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4349000000000001e-05 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 51/34 ECSubWrite(tid=60, reqid=client.4589.0:397, at_version=0'0, trim_to=0'0, roll_forward_to=51'30))", 2024-04-26T01:24:34.574 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.665670763, 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103627764, 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828851+0000", 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6000000000000004e-07 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828874+0000", 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3206000000000001e-05 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828896+0000", 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2209999999999999e-05 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828918+0000", 2024-04-26T01:24:34.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1687000000000001e-05 2024-04-26T01:24:34.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.929363+0000", 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100445169 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.929394+0000", 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0456000000000001e-05 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932323+0000", 2024-04-26T01:24:34.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029296819999999999 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932478+0000", 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015479400000000001 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/30 ECSubWrite(tid=97, reqid=client.4589.0:400, at_version=52'20, trim_to=0'0, roll_forward_to=51'18))", 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.562488144, 2024-04-26T01:24:34.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103051591, 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8599999999999999e-07 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932092+0000", 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8335999999999998e-05 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932116+0000", 2024-04-26T01:24:34.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4176e-05 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932131+0000", 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5187e-05 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932328+0000", 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019693900000000001 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932354+0000", 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5783e-05 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.035001+0000", 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102647556 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.035085+0000", 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3227999999999997e-05 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.584 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 52/34 ECSubWrite(tid=61, reqid=osd.11.0:61, at_version=52'32, trim_to=0'0, roll_forward_to=51'30))", 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.951351+0000", 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.543170109, 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102394516, 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.951351+0000", 2024-04-26T01:24:34.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.951351+0000", 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.951352+0000", 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.951386+0000", 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4153000000000003e-05 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.951407+0000", 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1163e-05 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.951427+0000", 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9256e-05 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.051653+0000", 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100226647 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.051683+0000", 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9696000000000002e-05 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.053660+0000", 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019773970000000001 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.053746+0000", 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5253999999999994e-05 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 53/34 ECSubWrite(tid=63, reqid=osd.11.0:63, at_version=53'34, trim_to=0'0, roll_forward_to=52'32))", 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:24.884726+0000", 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 9.6097949630000006, 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102783786, 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:34.589 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884726+0000", 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884726+0000", 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:34.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884727+0000", 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884784+0000", 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7509000000000001e-05 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:34.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884809+0000", 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4416000000000001e-05 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.984945+0000", 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100136112 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.985103+0000", 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015849399999999999 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.985133+0000", 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9413000000000001e-05 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.987420+0000", 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022869169999999999 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:34.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.987510+0000", 2024-04-26T01:24:34.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0235000000000006e-05 2024-04-26T01:24:34.594 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.594 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.594 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.594 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:34.594 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:34.594 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:34.596 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:34.596 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:34.599+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:34.696 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:34.701+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:34.766 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:34.766 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:34.766 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:34.766 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:34.775 INFO:teuthology.orchestra.run.smithi037.stderr:marked in osd.7. 2024-04-26T01:24:34.781 INFO:tasks.rados.rados.0.smithi181.stdout:190: done (2 left) 2024-04-26T01:24:34.781 INFO:tasks.rados.rados.0.smithi181.stdout:193: done (1 left) 2024-04-26T01:24:34.781 INFO:tasks.rados.rados.0.smithi181.stdout:195: done (0 left) 2024-04-26T01:24:34.781 INFO:tasks.rados.rados.0.smithi181.stdout:196: read oid 45 snap -1 2024-04-26T01:24:34.781 INFO:tasks.rados.rados.0.smithi181.stdout:196: expect (ObjNum 80 snap 13 seq_num 80) 2024-04-26T01:24:34.781 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 31 current snap is 15 2024-04-26T01:24:34.781 INFO:tasks.rados.rados.0.smithi181.stdout:197: seq_num 83 ranges {2867200=638976,3506176=622592,4128768=147456} 2024-04-26T01:24:34.783 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:34.797 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:34.799+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:34.804 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-04-26T01:24:34.828 INFO:tasks.rados.rados.0.smithi181.stdout:197: writing smithi18128752-31 from 2867200 to 3506176 tid 1 2024-04-26T01:24:34.835 INFO:tasks.rados.rados.0.smithi181.stdout:197: writing smithi18128752-31 from 3506176 to 4128768 tid 2 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:197: writing smithi18128752-31 from 4128768 to 4276224 tid 3 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:198: read oid 49 snap -1 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:198: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:199: read oid 38 snap -1 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:199: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:200: read oid 40 snap -1 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:200: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:24:34.838 INFO:tasks.rados.rados.0.smithi181.stdout:201: snap_create 2024-04-26T01:24:34.898 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:34.901+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:34.900 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:34.900 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:34.900 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:34.900 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:34.900 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:34.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:34.991 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:34.998 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:35.001+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:154 3.9s0 3:958f0a11:::smithi18128752-2:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:35.042 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.769289145999998, 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59153377200000001, 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 154 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:35.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274166+0000", 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.027e-06 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.044 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274180+0000", 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3723000000000001e-05 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274213+0000", 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3698999999999997e-05 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274229+0000", 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5928999999999999e-05 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274415+0000", 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018537499999999999 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274584+0000", 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016943000000000001 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662522+0000", 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38793827400000003 2024-04-26T01:24:35.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.664271+0000", 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00174884 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.865553+0000", 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20128183599999999 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.865697+0000", 2024-04-26T01:24:35.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014363900000000001 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:196 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.966052374, 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61790984400000004, 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.048 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 196 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8169999999999999e-06 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077416+0000", 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3783e-05 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7209000000000002e-05 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4168999999999998e-05 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077630+0000", 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015196599999999999 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077649+0000", 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9262999999999998e-05 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.181790+0000", 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10414122100000001 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182072+0000", 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00028194299999999999 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.183846+0000", 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001774184 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189006+0000", 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0051592809999999999 2024-04-26T01:24:35.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695163+0000", 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50615769499999996 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695310+0000", 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014631299999999999 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:197 3.9s0 3:93dca175:::smithi18128752-10:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.965923954000001, 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61806578000000001, 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 197 2024-04-26T01:24:35.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077529+0000", 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3239999999999999e-06 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077537+0000", 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0560000000000005e-06 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077556+0000", 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8456000000000001e-05 2024-04-26T01:24:35.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077573+0000", 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6566e-05 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077807+0000", 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023464800000000001 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077822+0000", 2024-04-26T01:24:35.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5206e-05 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184696+0000", 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10687403099999999 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184837+0000", 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000141113 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.186190+0000", 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001352575 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189008+0000", 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028181450000000002 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695495+0000", 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50648651300000003 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695594+0000", 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9147000000000005e-05 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:198 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.965857768999999, 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61826488800000001, 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.061 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 198 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077595+0000", 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.167e-06 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077601+0000", 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6500000000000001e-06 2024-04-26T01:24:35.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077619+0000", 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7569000000000001e-05 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077643+0000", 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4652999999999999e-05 2024-04-26T01:24:35.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077989+0000", 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034534799999999999 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078006+0000", 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7125999999999998e-05 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187076+0000", 2024-04-26T01:24:35.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10906990799999999 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187274+0000", 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000198221 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.188283+0000", 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001008994 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189285+0000", 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001002221 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695760+0000", 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.506474914 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695859+0000", 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9117000000000007e-05 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.067 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:211 3.0s0 3:014d0a8d:::smithi18128752-40:head [read 0~671744,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.078094+0000", 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.965358292000001, 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41330118300000002, 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.068 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 211 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078094+0000", 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078094+0000", 2024-04-26T01:24:35.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078095+0000", 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4699999999999999e-06 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078103+0000", 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1999999999999994e-06 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078122+0000", 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8445999999999999e-05 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078136+0000", 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4486e-05 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078272+0000", 2024-04-26T01:24:35.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013548100000000001 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078289+0000", 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7309e-05 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.083286+0000", 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0049972180000000003 2024-04-26T01:24:35.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.083443+0000", 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000156673 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.491395+0000", 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40795189999999998 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:312 3.9s0 3:958f0a11:::smithi18128752-2:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.614475+0000", 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.428977486000001, 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40588339800000001, 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 312 2024-04-26T01:24:35.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614475+0000", 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614475+0000", 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614476+0000", 2024-04-26T01:24:35.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614482+0000", 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.2299999999999996e-06 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.614499+0000", 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7028999999999999e-05 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714580+0000", 2024-04-26T01:24:35.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10008127999999999 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.716796+0000", 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002215561 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.716891+0000", 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4915999999999997e-05 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.020358+0000", 2024-04-26T01:24:35.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30346740900000002 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:313 3.0s0 3:03a7f525:::smithi18128752-19:head [append 0~671744] snapc 6=[4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.714620+0000", 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.328832340000002, 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51226638199999996, 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.078 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 313 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714620+0000", 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714620+0000", 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.714622+0000", 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.857e-06 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715165+0000", 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00054386400000000002 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715181+0000", 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5094e-05 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715191+0000", 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0074999999999999e-05 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.853370+0000", 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13817900999999999 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.854595+0000", 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012252820000000001 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.961048+0000", 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.106452781 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.961259+0000", 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000210919 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963480+0000", 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022215469999999999 2024-04-26T01:24:35.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969408+0000", 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0059276279999999999 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.226288+0000", 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25688034700000001 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.226886+0000", 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059797799999999999 2024-04-26T01:24:35.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:314 3.0s0 3:03a7f525:::smithi18128752-19:head [append 0~376832] snapc 6=[4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.328246480000001, 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51249058199999997, 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.084 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 314 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715206+0000", 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7300000000000005e-07 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715477+0000", 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027050499999999999 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715489+0000", 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2033e-05 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715497+0000", 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9799999999999998e-06 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.854744+0000", 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.139247491 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855021+0000", 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027672100000000001 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.965922+0000", 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.110901033 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.966294+0000", 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00037151900000000001 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968002+0000", 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001707914 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972283+0000", 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0042819019999999998 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.227207+0000", 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.25492373400000001 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.227696+0000", 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00048897700000000003 2024-04-26T01:24:35.089 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:315 3.0s0 3:03a7f525:::smithi18128752-19:head [setxattr _header (56) in=7b] snapc 6=[4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.327941427999999, 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53997301399999997, 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.090 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 315 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715515+0000", 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0879999999999997e-06 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715523+0000", 2024-04-26T01:24:35.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.605e-06 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715529+0000", 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1770000000000003e-06 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855140+0000", 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13961058700000001 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855264+0000", 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012369900000000001 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969747+0000", 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.114483211 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969923+0000", 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000175954 2024-04-26T01:24:35.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970982+0000", 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010588049999999999 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972408+0000", 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014263259999999999 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.254671+0000", 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28226294200000002 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.255484+0000", 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081285000000000005 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:316 3.0s0 3:03a7f525:::smithi18128752-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.327914902, 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86208679600000004, 2024-04-26T01:24:35.096 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 316 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715538+0000", 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.68e-07 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715540+0000", 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.413e-06 2024-04-26T01:24:35.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715547+0000", 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.131e-06 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715553+0000", 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6640000000000003e-06 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855355+0000", 2024-04-26T01:24:35.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13980195500000001 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855595+0000", 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000240265 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971741+0000", 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.116145744 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972105+0000", 2024-04-26T01:24:35.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036422899999999999 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272430+0000", 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30032516799999998 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272617+0000", 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018708799999999999 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.577624+0000", 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305006571 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:342 3.0s0 3:025aaed1:::smithi18128752-27:head [append 0~0] snapc 7=[4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:19.288497+0000", 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.754954677000001, 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58144454300000004, 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.102 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 342 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288497+0000", 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288497+0000", 2024-04-26T01:24:35.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288502+0000", 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2699999999999998e-06 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288518+0000", 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6220000000000001e-05 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288564+0000", 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6205999999999998e-05 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288589+0000", 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.525e-05 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.363623+0000", 2024-04-26T01:24:35.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.075033889000000006 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.363858+0000", 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023478600000000001 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.366695+0000", 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028370309999999998 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.372219+0000", 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0055238170000000003 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.869341+0000", 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49712220899999998 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.869942+0000", 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060086500000000001 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:400 3.9s0 3:9782f919:::smithi18128752-18:head [delete] snapc a=[9,8,4] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e52)", 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.776779+0000", 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.266673523, 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45556455299999998, 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.108 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 400 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.776779+0000", 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.109 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.776779+0000", 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.776782+0000", 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3569999999999998e-06 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.776795+0000", 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3526000000000001e-05 2024-04-26T01:24:35.110 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.776827+0000", 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1890000000000001e-05 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.776844+0000", 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6376e-05 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.877049+0000", 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100205561 2024-04-26T01:24:35.111 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.877062+0000", 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2333e-05 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.929523+0000", 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.052461530999999999 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.930037+0000", 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051409399999999999 2024-04-26T01:24:35.112 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.931935+0000", 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018974840000000001 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.935566+0000", 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036310320000000002 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.113 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.232117+0000", 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.296551167 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.232343+0000", 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000226202 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:438 3.0s0 3:0213ebcd:::smithi18128752-32:head [read 0~524288,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:28.779549+0000", 2024-04-26T01:24:35.114 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.2639032959999996, 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48895459899999999, 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 438 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.779549+0000", 2024-04-26T01:24:35.115 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.779549+0000", 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.779550+0000", 2024-04-26T01:24:35.116 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3769999999999999e-06 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.779578+0000", 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8296000000000001e-05 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.117 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.779595+0000", 2024-04-26T01:24:35.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6175e-05 2024-04-26T01:24:35.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.118 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.118 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.118 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.779610+0000", 2024-04-26T01:24:35.118 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5872999999999999e-05 2024-04-26T01:24:35.118 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.855270+0000", 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.075659757999999994 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.855290+0000", 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9456000000000001e-05 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.862870+0000", 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0075800620000000003 2024-04-26T01:24:35.119 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.863130+0000", 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000259798 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.268503+0000", 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.405373804 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.120 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(osd.4.21:9 3.0s0 3:0c3f65fa:::smithi18128752-8:head [assert-version v29,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.1602306489999998, 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79000656700000005, 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:24:35.121 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883225+0000", 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2959999999999999e-06 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.122 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883240+0000", 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4977000000000001e-05 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883273+0000", 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3818e-05 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883296+0000", 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2282000000000001e-05 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.123 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.368761+0000", 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48546481800000002 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.368922+0000", 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000161039 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.673228+0000", 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30430633699999998 2024-04-26T01:24:35.124 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:440 3.0s0 3:025aaed1:::smithi18128752-27:head [copy-from ver 12] snapc d=[c,8,4] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:29.781225+0000", 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 5.2622271820000002, 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.51731138300000001, 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.125 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 440 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781225+0000", 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781225+0000", 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.126 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781228+0000", 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0130000000000002e-06 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781242+0000", 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3626e-05 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781276+0000", 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4635999999999998e-05 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.127 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781301+0000", 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4405000000000001e-05 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781428+0000", 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012728800000000001 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.781440+0000", 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2489e-05 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.128 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:30.086120+0000", 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30467921599999997 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:30.086302+0000", 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000182635 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:30.096114+0000", 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0098122810000000008 2024-04-26T01:24:35.129 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:30.199443+0000", 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10332820600000001 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:30.298398+0000", 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.098955754000000007 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:30.298536+0000", 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013783400000000001 2024-04-26T01:24:35.130 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:468 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~589824] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.858294+0000", 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 2.1851583059999999, 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59054596400000003, 2024-04-26T01:24:35.131 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 468 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.858294+0000", 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.132 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.858294+0000", 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.858297+0000", 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3000000000000002e-06 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.859432+0000", 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001134475 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.133 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.859461+0000", 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9132000000000002e-05 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.859477+0000", 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5956000000000001e-05 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.134 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.941451+0000", 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.081974285999999993 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.941775+0000", 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032373599999999998 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.944167+0000", 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023926170000000001 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.135 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968585+0000", 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.024417726000000001 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.448719+0000", 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48013421499999998 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.448840+0000", 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012052099999999999 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.136 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:469 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~573440] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.867391+0000", 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 2.1760606999999998, 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58168562499999998, 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.137 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 469 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.867391+0000", 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.867391+0000", 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.138 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.867394+0000", 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3970000000000001e-06 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.868151+0000", 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00075715999999999999 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.868183+0000", 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1742999999999997e-05 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.139 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.868198+0000", 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5773e-05 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.946903+0000", 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.078704791999999996 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.947177+0000", 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027333499999999998 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.140 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.948983+0000", 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018060680000000001 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968588+0000", 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.019605043999999999 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449003+0000", 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48041582199999999 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.141 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449077+0000", 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3491000000000005e-05 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:470 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~196608] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870374+0000", 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 2.1730785410000002, 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57888931700000001, 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.142 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 470 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870374+0000", 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.143 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870374+0000", 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870376+0000", 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.413e-06 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.144 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870658+0000", 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000282188 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870690+0000", 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2169000000000003e-05 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870706+0000", 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5739999999999998e-05 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.145 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968816+0000", 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.098110003000000001 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968990+0000", 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017443100000000001 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.970055+0000", 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010643009999999999 2024-04-26T01:24:35.146 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.973981+0000", 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0039264759999999999 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449202+0000", 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.475220637 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.147 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449263+0000", 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0958999999999997e-05 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:471 3.0s0 3:01bb5e34:::smithi18128752-45:head [setxattr _header (57) in=7b] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 2.172719109, 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67904865400000003, 2024-04-26T01:24:35.148 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 471 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:24:35.149 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870734+0000", 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.124e-06 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.150 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870740+0000", 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4530000000000001e-06 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870751+0000", 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1126e-05 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870759+0000", 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4370000000000007e-06 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.151 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.971074+0000", 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100315001 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.971323+0000", 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024919900000000002 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972178+0000", 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00085481999999999995 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.152 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.976513+0000", 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.004335317 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.549610+0000", 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57309677199999998 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.549782+0000", 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017140499999999999 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.153 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:472 3.0s0 3:01bb5e34:::smithi18128752-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 2.17268075, 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83389453400000002, 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:35.154 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 472 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:35.155 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870772+0000", 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3699999999999999e-07 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870776+0000", 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.416e-06 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870786+0000", 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0652999999999999e-05 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:35.156 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870795+0000", 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4640000000000006e-06 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.974442+0000", 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10364701900000001 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.974615+0000", 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017313100000000001 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:35.157 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.550708+0000", 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57609289699999999 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.550827+0000", 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000119185 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.704666+0000", 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15383893200000001 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.158 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:35.159 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:35.159 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:35.160 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:35.160 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 1 to smithi18128752-31 2024-04-26T01:24:35.160 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:35.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:35.162 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:35.163+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:35.174 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:35.174 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:35.195 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:35.195 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:35.196 INFO:tasks.rados.rados.0.smithi181.stdout:198: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T01:24:35.213 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:35.213 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:35.231 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:35.232 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:35.252 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:35.253 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:35.261 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:35.259+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:35.273 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:35.273 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:35.292 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:35.292 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:35.296 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:35.296 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:35.310 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:35.310 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:35.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:35.332 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:35.332 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:35.342 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 2 to smithi18128752-31 2024-04-26T01:24:35.353 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:35.353 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:35.362 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:35.363+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:35.373 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:35.373 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:35.395 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:35.396 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:35.418 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:35.418 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:35.442 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:35.442 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:35.463 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:35.467+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:35.465 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:35.465 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:35.466 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:35.466 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:35.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:35.502 INFO:tasks.rados.rados.0.smithi181.stdout:196: expect (ObjNum 80 snap 13 seq_num 80) 2024-04-26T01:24:35.564 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:35.567+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:35.598 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 3 to smithi18128752-31 2024-04-26T01:24:35.598 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 4 to smithi18128752-31 2024-04-26T01:24:35.601 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:35.601 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:35.602 INFO:tasks.rados.rados.0.smithi181.stdout:200: expect (ObjNum 39 snap 0 seq_num 39) 2024-04-26T01:24:35.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:35.666 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:35.667+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:35.767 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:35.767+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:35.780 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:35.780 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:35.785 INFO:tasks.rados.rados.0.smithi181.stdout:196: done (5 left) 2024-04-26T01:24:35.785 INFO:tasks.rados.rados.0.smithi181.stdout:198: done (4 left) 2024-04-26T01:24:35.785 INFO:tasks.rados.rados.0.smithi181.stdout:200: done (3 left) 2024-04-26T01:24:35.786 INFO:tasks.rados.rados.0.smithi181.stdout:201: done (2 left) 2024-04-26T01:24:35.786 INFO:tasks.rados.rados.0.smithi181.stdout:202: snap_remove snap 13 2024-04-26T01:24:35.790 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:24:35.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:35.868 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:35.867+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:35.918 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:35.918 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:35.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:24:35.975 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:35.981+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:36.054 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:36.054 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:36.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:24:36.070 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:36.071+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:36.171 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:36.171+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:36.237 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:36.237 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:36.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:24:36.272 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:36.271+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:36.372 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:36.372 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:36.373 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:36.375+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:36.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:24:36.474 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:36.475+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:36.502 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:36.502 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:36.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:24:36.575 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:36.579+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:36.626 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:36.626 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:36.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:24:36.683 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:36.689+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:36.759 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:36.759 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:36.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:24:36.777 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:36.779+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:36.786 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-04-26T01:24:36.794 INFO:tasks.rados.rados.0.smithi181.stdout:202: done (2 left) 2024-04-26T01:24:36.794 INFO:tasks.rados.rados.0.smithi181.stdout:203: read oid 37 snap -1 2024-04-26T01:24:36.794 INFO:tasks.rados.rados.0.smithi181.stdout:203: expect (ObjNum 60 snap 5 seq_num 60) 2024-04-26T01:24:36.794 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 16 2024-04-26T01:24:36.795 INFO:tasks.rados.rados.0.smithi181.stdout:204: seq_num 84 ranges {32768=655360,688128=638976,1327104=131072} 2024-04-26T01:24:36.799 INFO:tasks.rados.rados.0.smithi181.stdout:204: writing smithi18128752-50 from 32768 to 688128 tid 1 2024-04-26T01:24:36.804 INFO:tasks.rados.rados.0.smithi181.stdout:204: writing smithi18128752-50 from 688128 to 1327104 tid 2 2024-04-26T01:24:36.805 INFO:tasks.rados.rados.0.smithi181.stdout:204: writing smithi18128752-50 from 1327104 to 1458176 tid 3 2024-04-26T01:24:36.805 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 12 current snap is 16 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 5 to smithi18128752-31 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 31 updating version 0 to 70 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 31 version 70 is already newer than 68 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 31 version 70 is already newer than 67 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 31 version 70 is already newer than 69 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 70 (ObjNum 83 snap 15 seq_num 83) dirty exists 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:197: left oid 31 (ObjNum 83 snap 15 seq_num 83) 2024-04-26T01:24:36.806 INFO:tasks.rados.rados.0.smithi181.stdout:199: expect (ObjNum 37 snap 0 seq_num 37) 2024-04-26T01:24:36.825 INFO:tasks.rados.rados.0.smithi181.stdout:205: seq_num 85 ranges {65536=671744,737280=770048} 2024-04-26T01:24:36.830 INFO:tasks.rados.rados.0.smithi181.stdout:205: writing smithi18128752-12 from 65536 to 737280 tid 1 2024-04-26T01:24:36.837 INFO:tasks.rados.rados.0.smithi181.stdout:205: writing smithi18128752-12 from 737280 to 1507328 tid 2 2024-04-26T01:24:36.837 INFO:tasks.rados.rados.0.smithi181.stdout:197: done (4 left) 2024-04-26T01:24:36.837 INFO:tasks.rados.rados.0.smithi181.stdout:199: done (3 left) 2024-04-26T01:24:36.837 INFO:tasks.rados.rados.0.smithi181.stdout:206: rmattr oid 10 current snap is 16 2024-04-26T01:24:36.837 INFO:tasks.rados.rados.0.smithi181.stdout:206: done (3 left) 2024-04-26T01:24:36.837 INFO:tasks.rados.rados.0.smithi181.stdout:207: snap_remove snap 15 2024-04-26T01:24:36.878 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:36.879+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:36.916 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:36.916 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:36.930 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:24:36.979 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:36.985+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:37.069 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:37.069 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:37.080 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:37.085+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:37.085 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:24:37.181 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:37.183+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:37.205 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:37.205 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:37.219 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:24:37.307 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:37.311+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:37.383 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:37.383+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:37.392 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:37.392 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:37.407 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:24:37.487 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:37.495 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:37.499+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:37.578 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:37.578 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:37.584 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:37.587+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:37.593 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:24:37.641 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:37.641 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:37.661 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:37.662 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:37.683 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:37.683 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:37.684 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:37.687+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:37.703 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:37.703 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:37.717 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:37.717 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:37.721 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:37.721 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:37.733 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:24:37.741 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:37.741 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:37.762 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:37.762 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:37.784 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:37.785 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:37.785 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:37.787+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:37.803 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:37.804 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:37.826 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:37.826 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:37.848 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:37.848 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:37.853 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:122 3.ds0 3:b6ed1d05:::smithi18128752-41:head [setxattr _header (56) in=63b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.493522+0000", 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.363209292999997, 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59216864499999999, 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.854 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 122 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493522+0000", 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.855 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493522+0000", 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493523+0000", 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2399999999999998e-07 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493526+0000", 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3359999999999999e-06 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.856 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493532+0000", 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6239999999999999e-06 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493536+0000", 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3760000000000001e-06 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.696585+0000", 2024-04-26T01:24:37.857 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20304873200000001 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.696668+0000", 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.3269000000000005e-05 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697063+0000", 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00039460100000000001 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.858 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697824+0000", 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00076119200000000005 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.085525+0000", 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.38770083599999999 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.085691+0000", 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016605499999999999 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.859 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:123 3.ds0 3:b6ed1d05:::smithi18128752-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.363187568999997, 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.401407085, 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.860 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 123 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.861 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.862 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.862 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.862 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493545+0000", 2024-04-26T01:24:37.862 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5299999999999999e-07 2024-04-26T01:24:37.862 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.862 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.862 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493546+0000", 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7039999999999999e-06 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493552+0000", 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5119999999999999e-06 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493556+0000", 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2679999999999996e-06 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.863 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697434+0000", 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20387831000000001 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697559+0000", 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012467700000000001 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087644+0000", 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39008522000000001 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.864 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087733+0000", 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9191999999999996e-05 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894951+0000", 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80721774899999998 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:138 3.1s0 3:8830b48d:::smithi18128752-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:37.865 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.077309763999999, 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71477871400000004, 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 138 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.866 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779423+0000", 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.47e-07 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.867 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779429+0000", 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6400000000000002e-06 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779443+0000", 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4392999999999999e-05 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779456+0000", 2024-04-26T01:24:37.868 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2643000000000001e-05 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784323+0000", 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048675369999999999 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784379+0000", 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6314000000000003e-05 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.869 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187338+0000", 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40295840700000002 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187417+0000", 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8898000000000003e-05 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.494201+0000", 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30678393500000001 2024-04-26T01:24:37.870 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:139 3.ds0 3:b9916085:::smithi18128752-47:head [append 0~786432,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.101917+0000", 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.754814768999999, 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59102550499999995, 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.871 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 139 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.101917+0000", 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.872 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.101917+0000", 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.101919+0000", 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3889999999999999e-06 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.873 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102817+0000", 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00089738299999999997 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102843+0000", 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6627000000000001e-05 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102855+0000", 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1362000000000001e-05 2024-04-26T01:24:37.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.288872+0000", 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18601710499999999 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289007+0000", 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013478900000000001 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289905+0000", 2024-04-26T01:24:37.875 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00089878899999999999 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.492069+0000", 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20216367399999999 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.692795+0000", 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20072556799999999 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.876 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.692942+0000", 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014781899999999999 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:140 3.ds0 3:b9916085:::smithi18128752-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.753854867000001, 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.69088028800000001, 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.877 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 140 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.878 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102878+0000", 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2399999999999997e-07 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.879 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102881+0000", 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1489999999999998e-06 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102885+0000", 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1300000000000003e-06 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102888+0000", 2024-04-26T01:24:37.880 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.242e-06 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290947+0000", 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18805879 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291087+0000", 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013990199999999999 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.881 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291629+0000", 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054169199999999998 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.492101+0000", 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20047221400000001 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.793519+0000", 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30141857300000002 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.882 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.793757+0000", 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000237772 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:141 3.ds0 3:b9916085:::smithi18128752-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.753838695999999, 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.99325914900000001, 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.883 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 141 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.884 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102896+0000", 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3209999999999999e-06 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102900+0000", 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9369999999999997e-06 2024-04-26T01:24:37.885 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102903+0000", 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3809999999999999e-06 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291929+0000", 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.189026048 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.292027+0000", 2024-04-26T01:24:37.886 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8172999999999997e-05 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794279+0000", 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50225164799999999 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794416+0000", 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013710400000000001 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.887 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.096152+0000", 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30173626199999998 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/30 ECSubReadReply(tid=43, attrs_read=0))", 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.288989+0000", 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.567742251999999, 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60597367199999996, 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:37.888 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.288989+0000", 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.288989+0000", 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.288991+0000", 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.133e-06 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.889 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289001+0000", 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0016e-05 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289010+0000", 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1700000000000003e-06 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289035+0000", 2024-04-26T01:24:37.890 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4893e-05 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894810+0000", 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60577542399999995 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894963+0000", 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000153036 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.891 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:325 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.495406125999999, 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70936785199999997, 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 325 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.892 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361328+0000", 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3580000000000001e-06 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.893 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361338+0000", 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.628e-06 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361360+0000", 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2506000000000001e-05 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361374+0000", 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4072e-05 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.894 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454039+0000", 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.092664721000000005 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454298+0000", 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025867499999999999 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.455951+0000", 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016533489999999999 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.895 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462924+0000", 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069735650000000001 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070553+0000", 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60762835599999998 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070693+0000", 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000140622 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.896 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:326 3.7s0 3:e977e128:::smithi18128752-21:head [rollback 4] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.495337559999999, 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70968584099999998, 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 326 2024-04-26T01:24:37.897 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.898 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361395+0000", 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.06e-07 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361398+0000", 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1379999999999999e-06 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361408+0000", 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.046e-05 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.899 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361414+0000", 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8459999999999996e-06 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458365+0000", 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096950878000000004 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458663+0000", 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000297788 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.900 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.460203+0000", 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015404030000000001 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.463428+0000", 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032243850000000002 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.901 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070927+0000", 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.607499821 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071080+0000", 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000152516 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:327 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:37.902 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.495309733999999, 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71007962499999999, 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 327 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.903 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.904 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361425+0000", 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2740000000000002e-06 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361431+0000", 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4280000000000001e-06 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361436+0000", 2024-04-26T01:24:37.905 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.994e-06 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461250+0000", 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099813814000000001 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461584+0000", 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000334521 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.906 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462509+0000", 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00092412699999999996 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.565364+0000", 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102855424 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071318+0000", 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50595366500000005 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071502+0000", 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000183912 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:328 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.495289329999999, 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81248642000000004, 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.908 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 328 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.909 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361443+0000", 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6199999999999998e-07 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361447+0000", 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.262e-06 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.910 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361454+0000", 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8739999999999997e-06 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361459+0000", 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8040000000000001e-06 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563609+0000", 2024-04-26T01:24:37.911 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20214983 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563788+0000", 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017973199999999999 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.173929+0000", 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.610140456 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.912 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:329 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.495266734000001, 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81314448500000003, 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 329 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.913 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2399999999999999e-07 2024-04-26T01:24:37.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361469+0000", 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2820000000000001e-06 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361475+0000", 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3509999999999998e-06 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361480+0000", 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6959999999999996e-06 2024-04-26T01:24:37.915 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564389+0000", 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202908808 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564593+0000", 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000204255 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.174609+0000", 2024-04-26T01:24:37.916 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.61001666899999996 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:330 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.495246222999999, 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81407258199999999, 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.917 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 330 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361486+0000", 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0999999999999999e-07 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361489+0000", 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7240000000000001e-06 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361495+0000", 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2740000000000004e-06 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361500+0000", 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9400000000000001e-06 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.866935+0000", 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50543468599999997 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.867233+0000", 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029877499999999998 2024-04-26T01:24:37.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.175558+0000", 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.308324773 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.7s0 47/30 ECSubReadReply(tid=129, attrs_read=0))", 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.566298+0000", 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 19.290433408999998, 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50652343399999999, 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.921 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.566298+0000", 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.566298+0000", 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.566299+0000", 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1299999999999999e-07 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.566628+0000", 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00032847799999999999 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.566645+0000", 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7195999999999999e-05 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.923 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.666751+0000", 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100106056 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.072719+0000", 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40596789599999999 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.072822+0000", 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000102995 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:434 3.7s0 3:e9e07e85:::smithi18128752-17:head [copy-from ver 29 in=54b] snapc d=[d,c,8,4] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.0780795439999995, 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1051031140000001, 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 434 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.925 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778655+0000", 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8770000000000002e-06 2024-04-26T01:24:37.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778667+0000", 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1888999999999999e-05 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778697+0000", 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0205999999999998e-05 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778714+0000", 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6467999999999999e-05 2024-04-26T01:24:37.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878928+0000", 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100213993 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878940+0000", 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2724999999999999e-05 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882181+0000", 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032411419999999998 2024-04-26T01:24:37.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882351+0000", 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169371 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.678907+0000", 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.79655591199999998 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.882986+0000", 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.204079063 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883081+0000", 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5285000000000007e-05 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883755+0000", 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00067418299999999997 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.930 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:435 3.7s0 3:e9e07e85:::smithi18128752-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 9.0779838870000003, 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.003729404, 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 435 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.931 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778749+0000", 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.223e-06 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778755+0000", 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8279999999999999e-06 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778773+0000", 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8289000000000001e-05 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878906+0000", 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10013274699999999 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.933 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879035+0000", 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012864199999999999 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879049+0000", 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4137e-05 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882650+0000", 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036017390000000001 2024-04-26T01:24:37.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882716+0000", 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5129000000000006e-05 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782226+0000", 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.89951035899999998 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782341+0000", 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000115492 2024-04-26T01:24:37.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782477+0000", 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000135819 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:474 3.bs0 3:d65fea44:::smithi18128752-20:head [append 0~720896] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.891691+0000", 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.9650407770000005, 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51016994000000004, 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.936 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 474 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.891691+0000", 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.937 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.891691+0000", 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.891692+0000", 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.421e-06 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892418+0000", 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00072603900000000005 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.938 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892436+0000", 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7963999999999999e-05 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892446+0000", 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6069999999999997e-06 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.988384+0000", 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.095938332000000001 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.939 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.988523+0000", 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013893999999999999 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.989569+0000", 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001046104 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992362+0000", 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0027927899999999999 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.401738+0000", 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40937615900000002 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.401861+0000", 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012258399999999999 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.941 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:475 3.bs0 3:d65fea44:::smithi18128752-20:head [append 0~49152] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.892478+0000", 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.9642534329999997, 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50972112599999997, 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 475 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.942 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892478+0000", 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892478+0000", 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892480+0000", 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3370000000000001e-06 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.943 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892529+0000", 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9651000000000003e-05 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892540+0000", 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0859999999999999e-05 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892548+0000", 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7000000000000008e-06 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.944 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.990536+0000", 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.097988200999999997 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.990651+0000", 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011523699999999999 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.991229+0000", 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00057820199999999999 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.945 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992365+0000", 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001135795 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402086+0000", 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40972029500000001 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402199+0000", 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113848 2024-04-26T01:24:37.946 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:476 3.bs0 3:d65fea44:::smithi18128752-20:head [setxattr _header (57) in=7b] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.892559+0000", 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.9641729559999996, 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50988418499999999, 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.947 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 476 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892559+0000", 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892559+0000", 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892559+0000", 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7399999999999998e-07 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892563+0000", 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2699999999999998e-06 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892569+0000", 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1340000000000003e-06 2024-04-26T01:24:37.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892573+0000", 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1439999999999996e-06 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.991666+0000", 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099092871999999999 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.991726+0000", 2024-04-26T01:24:37.950 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0619999999999999e-05 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992064+0000", 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000337306 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992931+0000", 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00086702700000000003 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:37.951 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402338+0000", 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40940727199999999 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402443+0000", 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000105066 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:477 3.bs0 3:d65fea44:::smithi18128752-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.9641522479999995, 2024-04-26T01:24:37.952 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.91386205600000003, 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 477 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892580+0000", 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7999999999999996e-07 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892582+0000", 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9630000000000001e-06 2024-04-26T01:24:37.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892589+0000", 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7959999999999998e-06 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892593+0000", 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2509999999999996e-06 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992304+0000", 2024-04-26T01:24:37.955 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099710909 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992356+0000", 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2021000000000002e-05 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402928+0000", 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.41057238600000001 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.403133+0000", 2024-04-26T01:24:37.956 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020520099999999999 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.806441+0000", 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.403308049 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:37.957 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:37.960 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:24:37.960 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:37.960 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:207: done (3 left) 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 2 current snap is 16 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:208: seq_num 86 ranges {180224=688128,868352=540672,1409024=229376} 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:208: writing smithi18128752-2 from 180224 to 868352 tid 1 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:208: writing smithi18128752-2 from 868352 to 1409024 tid 2 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:208: writing smithi18128752-2 from 1409024 to 1638400 tid 3 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:209: copy_from oid 38 from oid 46 current snap is 16 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:204: finishing write tid 1 to smithi18128752-50 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:204: finishing write tid 2 to smithi18128752-50 2024-04-26T01:24:37.961 INFO:tasks.rados.rados.0.smithi181.stdout:203: expect (ObjNum 60 snap 5 seq_num 60) 2024-04-26T01:24:37.962 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:37.962 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:37.962 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:37.962 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:37.963 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:37.963 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:37.963 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:37.963+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:37.967 INFO:tasks.rados.rados.0.smithi181.stdout:204: finishing write tid 3 to smithi18128752-50 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:205: finishing write tid 1 to smithi18128752-12 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: finishing write tid 4 to smithi18128752-50 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:205: finishing write tid 2 to smithi18128752-12 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: finishing write tid 5 to smithi18128752-50 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: oid 50 updating version 0 to 49 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: oid 50 version 49 is already newer than 46 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: oid 50 version 49 is already newer than 47 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: oid 50 version 49 is already newer than 48 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 49 (ObjNum 84 snap 16 seq_num 84) dirty exists 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: left oid 50 (ObjNum 84 snap 16 seq_num 84) 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:205: finishing write tid 3 to smithi18128752-12 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:203: done (4 left) 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:204: done (3 left) 2024-04-26T01:24:37.968 INFO:tasks.rados.rados.0.smithi181.stdout:210: snap_remove snap 2 2024-04-26T01:24:38.001 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:38.002 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:38.067 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:38.071+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:38.141 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:38.141 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:38.142 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:38.142 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:38.156 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:24:38.161 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:38.165+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:38.262 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:38.263+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:38.321 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:38.515 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:38.516 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:38.516 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:38.520 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:38.519+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:38.521 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:24:38.620 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:38.623+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 35/34 ECSubWrite(tid=13, reqid=client.4589.0:74, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.608020121000003, 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20101313800000001, 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.660 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054192+0000", 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054193+0000", 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.13e-06 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054208+0000", 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4562000000000001e-05 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.054223+0000", 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5536000000000002e-05 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.154317+0000", 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100094023 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.254494+0000", 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100176788 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.254508+0000", 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3779e-05 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255170+0000", 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00066210200000000002 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.255205+0000", 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5218000000000001e-05 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.664 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=16, reqid=client.4589.0:86, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.475200713, 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.1988666, 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187011+0000", 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187013+0000", 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8840000000000001e-06 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187073+0000", 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9972000000000001e-05 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187121+0000", 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7222999999999998e-05 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187151+0000", 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0107000000000002e-05 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284476+0000", 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.097325196000000003 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.284487+0000", 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1633e-05 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385837+0000", 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10134913199999999 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385878+0000", 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1452999999999998e-05 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=17, reqid=client.4589.0:146, at_version=35'9, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.471833328999999, 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19552889900000001, 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.669 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190379+0000", 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190380+0000", 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5099999999999999e-06 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190583+0000", 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020251199999999999 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190609+0000", 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6256999999999999e-05 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.190624+0000", 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5302999999999999e-05 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.384789+0000", 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.194164533 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.384800+0000", 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0919e-05 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385883+0000", 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001082902 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385908+0000", 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4963000000000001e-05 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:38.673 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.469028770999998, 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19307022800000001, 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193183+0000", 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193185+0000", 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193366+0000", 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000181096 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193393+0000", 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6673e-05 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.193408+0000", 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.554e-05 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385391+0000", 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19198310199999999 2024-04-26T01:24:38.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.385402+0000", 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0586e-05 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.386222+0000", 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081973500000000004 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.386254+0000", 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1866000000000003e-05 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 35/34 ECSubWrite(tid=19, reqid=client.4589.0:148, at_version=35'11, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.496969+0000", 2024-04-26T01:24:38.678 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.165243570999998, 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20146192700000001, 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.496969+0000", 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.496969+0000", 2024-04-26T01:24:38.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.496970+0000", 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8029999999999999e-06 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497037+0000", 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6655999999999993e-05 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497066+0000", 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8663e-05 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497084+0000", 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8430000000000001e-05 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.697353+0000", 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20026850900000001 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.681 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.697368+0000", 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4970000000000001e-05 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.698370+0000", 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010020070000000001 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.698431+0000", 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0888999999999999e-05 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.682 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:185 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.589000997999999, 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.395059522, 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 185 2024-04-26T01:24:38.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073214+0000", 2024-04-26T01:24:38.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073225+0000", 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0404999999999999e-05 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073253+0000", 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8058000000000001e-05 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173330+0000", 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10007712000000001 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.261568+0000", 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.088237503999999994 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.261653+0000", 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5259999999999993e-05 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.262369+0000", 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00071622000000000005 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265729+0000", 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003360264 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.468152+0000", 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20242310199999999 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.468271+0000", 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000118279 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(osd.4.21:2 3.4s0 3:2e10a639:::smithi18128752-28:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.502123558000001, 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30734204100000001, 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.688 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160090+0000", 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7129999999999999e-06 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160104+0000", 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3596e-05 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160114+0000", 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8190000000000003e-06 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160125+0000", 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1413999999999999e-05 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160185+0000", 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9914000000000003e-05 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160251+0000", 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5728999999999994e-05 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467431+0000", 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30717985599999997 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:186 3.4s0 3:2be1e129:::smithi18128752-30:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.488864883000002, 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39528098299999997, 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 186 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.693 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173348+0000", 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.108e-06 2024-04-26T01:24:38.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173354+0000", 2024-04-26T01:24:38.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3009999999999997e-06 2024-04-26T01:24:38.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173362+0000", 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9710000000000004e-06 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173367+0000", 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3179999999999997e-06 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263197+0000", 2024-04-26T01:24:38.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.089829862999999996 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263286+0000", 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9519999999999997e-05 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263786+0000", 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00050013300000000004 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265873+0000", 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002086606 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568417+0000", 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30254369399999997 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568628+0000", 2024-04-26T01:24:38.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021146899999999999 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:187 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.488837664999998, 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51780501899999998, 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.699 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 187 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173375+0000", 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1699999999999999e-07 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173377+0000", 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.939e-06 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173381+0000", 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2050000000000004e-06 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173384+0000", 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2760000000000001e-06 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264258+0000", 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.090873535000000005 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264532+0000", 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027448400000000002 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265351+0000", 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081914900000000003 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.466871+0000", 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20151915000000001 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690961+0000", 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22409068500000001 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.691180+0000", 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021827899999999999 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 37/30 ECSubWriteReply(tid=14, last_complete=37'9, committed=1, applied=1))", 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.397531971999999, 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.203106276, 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.705 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264681+0000", 2024-04-26T01:24:38.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5999999999999991e-07 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264687+0000", 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9549999999999997e-06 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264702+0000", 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.486e-05 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.364796+0000", 2024-04-26T01:24:38.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100093734 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467752+0000", 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102956666 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467786+0000", 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4100999999999998e-05 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 37/30 ECSubWriteReply(tid=15, last_complete=37'10, committed=1, applied=1))", 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.490328+0000", 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 30.171883707999999, 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20036198099999999, 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490328+0000", 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490328+0000", 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490330+0000", 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9470000000000002e-06 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490340+0000", 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6870000000000006e-06 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.490370+0000", 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9952000000000001e-05 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.590485+0000", 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10011507 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690653+0000", 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100167637 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690690+0000", 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7688000000000002e-05 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 40/34 ECSubWrite(tid=40, reqid=client.4589.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=40'23))", 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.008805+0000", 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.653406797999999, 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18152664700000001, 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.712 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008805+0000", 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008805+0000", 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008807+0000", 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.734e-06 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008833+0000", 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5772999999999999e-05 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008862+0000", 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9003e-05 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.008879+0000", 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6699999999999999e-05 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.089652+0000", 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.080773257000000001 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.089677+0000", 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5185999999999999e-05 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.190283+0000", 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100606271 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.190332+0000", 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8723000000000001e-05 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 40/30 ECSubWriteReply(tid=20, last_complete=41'12, committed=1, applied=1))", 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.093963+0000", 2024-04-26T01:24:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.568248672999999, 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20213109300000001, 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093963+0000", 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093963+0000", 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093966+0000", 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2000000000000001e-06 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.093977+0000", 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1226000000000001e-05 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.094000+0000", 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3516999999999999e-05 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.194129+0000", 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100128113 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296007+0000", 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101878477 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296095+0000", 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7559999999999995e-05 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:270 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.489298673, 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22773885299999999, 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.720 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 270 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:38.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172916+0000", 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0170000000000001e-06 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172932+0000", 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5766000000000001e-05 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172963+0000", 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0440999999999999e-05 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172980+0000", 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7224e-05 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.295355+0000", 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.122375555 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:38.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.295455+0000", 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9192999999999997e-05 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.400652+0000", 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.105197657 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:271 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:38.724 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.489196201999999, 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53167738200000003, 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 271 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173017+0000", 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1100000000000005e-07 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173022+0000", 2024-04-26T01:24:38.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4990000000000002e-06 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173034+0000", 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1667e-05 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173044+0000", 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9070000000000002e-06 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398177+0000", 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225132687 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398388+0000", 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000211325 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.704693+0000", 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30630548600000002 2024-04-26T01:24:38.728 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:272 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.489154523, 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53273070499999997, 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:38.729 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 272 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9200000000000001e-07 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173079+0000", 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1137000000000001e-05 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173093+0000", 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3198e-05 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173102+0000", 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1859999999999998e-06 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401582+0000", 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22848043000000001 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401813+0000", 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023127900000000001 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.705788+0000", 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30397488299999997 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 41/34 ECSubWrite(tid=33, reqid=osd.11.0:33, at_version=41'13, trim_to=0'0, roll_forward_to=37'11))", 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.206421+0000", 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.455790849, 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.19296364399999999, 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.733 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206421+0000", 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206421+0000", 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206422+0000", 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3e-07 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206517+0000", 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4803999999999996e-05 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206549+0000", 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2456000000000002e-05 2024-04-26T01:24:38.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.206574+0000", 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4453000000000001e-05 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296181+0000", 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.089607645 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.296201+0000", 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9967999999999999e-05 2024-04-26T01:24:38.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.399316+0000", 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103114385 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.399385+0000", 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.9302999999999997e-05 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.4s0 41/30 ECSubRead(tid=24, to_read={3:2e10a639:::smithi18128752-28:head=0,114688,0}, subchunks={3:2e10a639:::smithi18128752-28:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 26.259887491000001, 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20080430399999999, 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402325+0000", 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.402377+0000", 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2645999999999997e-05 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.502485+0000", 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100107906 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.602850+0000", 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100364604 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.603129+0000", 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000279148 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s4 47/30 ECSubWrite(tid=126, reqid=client.4589.0:327, at_version=47'30, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.462580+0000", 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 20.199632270999999, 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201538619, 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.741 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462580+0000", 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462580+0000", 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462582+0000", 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6640000000000001e-06 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462626+0000", 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4576000000000001e-05 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462667+0000", 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0769000000000002e-05 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462690+0000", 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2900000000000001e-05 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.663034+0000", 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20034379699999999 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.663052+0000", 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8717000000000001e-05 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.664050+0000", 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00099748999999999997 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.664118+0000", 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8706000000000003e-05 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.745 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s4 61/30 ECSubWrite(tid=177, reqid=client.4589.0:478, at_version=61'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.899783+0000", 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.7624287609999998, 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20341484900000001, 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899783+0000", 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.746 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899783+0000", 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899785+0000", 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3230000000000001e-06 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899883+0000", 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7841000000000001e-05 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899904+0000", 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.105e-05 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.999981+0000", 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10007708899999999 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.100182+0000", 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100201789 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.100210+0000", 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7166e-05 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.103164+0000", 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0029540679999999998 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.103198+0000", 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4523000000000001e-05 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.749 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.750 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.750 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:38.750 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:38.750 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:38.751 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:24:38.752 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:38.755+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:38.776 INFO:tasks.rados.rados.0.smithi181.stdout:210: done (3 left) 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:211: rmattr oid 8 current snap is 16 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:211: done (3 left) 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:212: rmattr oid 25 current snap is 16 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:212: done (3 left) 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:213: rmattr oid 4 current snap is 16 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:209: finishing copy_from racing read to smithi18128752-38 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:208: finishing write tid 1 to smithi18128752-2 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:205: finishing write tid 4 to smithi18128752-12 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:205: oid 12 updating version 0 to 18 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:205: oid 12 updating version 18 to 19 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:205: oid 12 version 19 is already newer than 17 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 19 (ObjNum 85 snap 16 seq_num 85) dirty exists 2024-04-26T01:24:38.777 INFO:tasks.rados.rados.0.smithi181.stdout:205: left oid 12 (ObjNum 85 snap 16 seq_num 85) 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:209: finishing copy_from to smithi18128752-38 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 7 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: finishing write tid 2 to smithi18128752-2 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: finishing write tid 3 to smithi18128752-2 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: finishing write tid 4 to smithi18128752-2 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: finishing write tid 5 to smithi18128752-2 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: oid 2 updating version 0 to 26 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: oid 2 updating version 26 to 27 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: oid 2 updating version 27 to 29 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: oid 2 version 29 is already newer than 28 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 29 (ObjNum 86 snap 16 seq_num 86) dirty exists 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: left oid 2 (ObjNum 86 snap 16 seq_num 86) 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:205: done (3 left) 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:208: done (2 left) 2024-04-26T01:24:38.778 INFO:tasks.rados.rados.0.smithi181.stdout:209: done (1 left) 2024-04-26T01:24:38.779 INFO:tasks.rados.rados.0.smithi181.stdout:214: read oid 5 snap -1 2024-04-26T01:24:38.779 INFO:tasks.rados.rados.0.smithi181.stdout:214: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:38.779 INFO:tasks.rados.rados.0.smithi181.stdout:215: read oid 48 snap -1 2024-04-26T01:24:38.779 INFO:tasks.rados.rados.0.smithi181.stdout:215: expect (ObjNum 79 snap 12 seq_num 79) 2024-04-26T01:24:38.779 INFO:tasks.rados.rados.0.smithi181.stdout:216: setattr oid 38 current snap is 16 2024-04-26T01:24:38.782 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 37 current snap is 16 2024-04-26T01:24:38.782 INFO:tasks.rados.rados.0.smithi181.stdout:217: seq_num 87 ranges {1622016=704512,2326528=688128,3014656=81920} 2024-04-26T01:24:38.815 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:24:38.820 INFO:tasks.rados.rados.0.smithi181.stdout:217: writing smithi18128752-37 from 1622016 to 2326528 tid 1 2024-04-26T01:24:38.830 INFO:tasks.rados.rados.0.smithi181.stdout:217: writing smithi18128752-37 from 2326528 to 3014656 tid 2 2024-04-26T01:24:38.831 INFO:tasks.rados.rados.0.smithi181.stdout:217: writing smithi18128752-37 from 3014656 to 3096576 tid 3 2024-04-26T01:24:38.832 INFO:tasks.rados.rados.0.smithi181.stdout:218: copy_from oid 41 from oid 31 current snap is 16 2024-04-26T01:24:38.832 INFO:tasks.rados.rados.0.smithi181.stdout:219: snap_create 2024-04-26T01:24:38.852 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:38.855+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:38.936 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:38.936 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:38.936 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:38.936 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:38.951 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:24:38.953 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:38.955+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:39.055 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:39.059+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:39.078 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:39.078 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:39.078 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:39.092 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:24:39.155 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:39.159+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.6s4 35/34 ECSubRead(tid=8, to_read={3:60f0095a:::smithi18128752-3:head=0,262144,0}, subchunks={3:60f0095a:::smithi18128752-3:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 32.801372872000002, 2024-04-26T01:24:39.216 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100734536, 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:39.217 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420052+0000", 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.846e-06 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420069+0000", 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7493e-05 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420098+0000", 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9079000000000001e-05 2024-04-26T01:24:39.218 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420118+0000", 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9468999999999999e-05 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520369+0000", 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100251189 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520785+0000", 2024-04-26T01:24:39.219 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00041545999999999998 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.es6 36/34 ECSubRead(tid=24, to_read={3:738de3f3:::smithi18128752-9:head=0,262144,0}, subchunks={3:738de3f3:::smithi18128752-9:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.501681172000001, 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00074145500000000002, 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.220 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.03e-07 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.221 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719756+0000", 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4053000000000001e-05 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719789+0000", 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2913999999999997e-05 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719810+0000", 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0513000000000002e-05 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.222 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719965+0000", 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015534199999999999 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.720483+0000", 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051822999999999995 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:39.223 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:39.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:24:39.232 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 46 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-04-26T01:24:39.256 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:39.259+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:39.330 INFO:tasks.rados.rados.0.smithi181.stdout:214: expect (ObjNum 4 snap 0 seq_num 4) 2024-04-26T01:24:39.338 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 8 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-26T01:24:39.340 INFO:tasks.rados.rados.0.smithi181.stdout:215: expect (ObjNum 79 snap 12 seq_num 79) 2024-04-26T01:24:39.357 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:39.359+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:39.366 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s2 67/63 ECSubWrite(tid=51, reqid=osd.0.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=67'33))", 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.234680+0000", 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.13734853, 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.13738915600000001, 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.367 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234680+0000", 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234680+0000", 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234681+0000", 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.097e-06 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234705+0000", 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3405e-05 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234733+0000", 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8663999999999999e-05 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.334874+0000", 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100140251 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.369 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 67/44 ECSubWrite(tid=263, reqid=osd.13.0:263, at_version=67'51, trim_to=0'0, roll_forward_to=67'49))", 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.305228+0000", 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.066800595000000004, 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.066931209000000005, 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.305228+0000", 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.305228+0000", 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.305228+0000", 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1399999999999998e-07 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.305257+0000", 2024-04-26T01:24:39.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8799e-05 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.305267+0000", 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8099999999999992e-06 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.305276+0000", 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0380000000000005e-06 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.372 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/44 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=67'64, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.037010772999999997, 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.037219452, 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.373 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3700000000000001e-07 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335038+0000", 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0047e-05 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335047+0000", 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7329999999999992e-06 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.375 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.376 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:39.376 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 3 2024-04-26T01:24:39.376 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:39.381 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:24:39.458 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:39.459+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:39.551 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:39.551 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:39.551 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:39.551 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:39.551 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:39.558 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:39.561+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:39.565 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:24:39.629 INFO:tasks.rados.rados.0.smithi181.stdout:218: finishing copy_from racing read to smithi18128752-41 2024-04-26T01:24:39.658 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:39.663+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:39.694 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=11, reqid=client.4589.0:127, at_version=35'7, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:39.694 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.118695017, 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202990695, 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:39.695 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577859+0000", 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.013e-06 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578257+0000", 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039834700000000001 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.696 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578289+0000", 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2016e-05 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578306+0000", 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6460000000000002e-05 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678641+0000", 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100335199 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.697 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678656+0000", 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5233e-05 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780779+0000", 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102122955 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780847+0000", 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8472000000000004e-05 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.698 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=31, reqid=client.4589.0:124, at_version=35'13, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.845173764000002, 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20482444299999999, 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.699 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851380+0000", 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.973e-06 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852052+0000", 2024-04-26T01:24:39.700 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067198400000000004 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852084+0000", 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2298999999999997e-05 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952200+0000", 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100116018 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952352+0000", 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015207299999999999 2024-04-26T01:24:39.701 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952377+0000", 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4363e-05 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056116+0000", 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103738862 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056202+0000", 2024-04-26T01:24:39.702 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6871000000000002e-05 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 35/30 ECSubWrite(tid=170, reqid=client.4589.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.727237275999997, 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20178349000000001, 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.703 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969317+0000", 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0600000000000002e-06 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.704 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969365+0000", 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8296e-05 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969439+0000", 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3907999999999998e-05 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969466+0000", 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7223e-05 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.705 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169873+0000", 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20040670999999999 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169887+0000", 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4727e-05 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171018+0000", 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001130968 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.706 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171098+0000", 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9598000000000006e-05 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.419466489999998, 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20597036399999999, 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.707 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.708 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277086+0000", 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277250+0000", 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016446899999999999 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277265+0000", 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.418e-05 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.709 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277275+0000", 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0456e-05 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481015+0000", 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203739686 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481042+0000", 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6962999999999999e-05 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.710 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482955+0000", 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001913816 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483056+0000", 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100204 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.4s3 41/30 ECSubWrite(tid=22, reqid=osd.8.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-04-26T01:24:39.711 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 27.299999814, 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20184675299999999, 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.712 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396553+0000", 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.31e-06 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396576+0000", 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2816e-05 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396597+0000", 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1166999999999999e-05 2024-04-26T01:24:39.713 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396611+0000", 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4099e-05 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.496984+0000", 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100372218 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.497002+0000", 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8051999999999999e-05 2024-04-26T01:24:39.714 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.598348+0000", 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10134613200000001 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.598399+0000", 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0958999999999998e-05 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 41/34 ECSubWrite(tid=45, reqid=osd.2.0:44, at_version=0'0, trim_to=0'0, roll_forward_to=41'27))", 2024-04-26T01:24:39.715 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.597646+0000", 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 27.098905430999999, 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20104497700000001, 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597646+0000", 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597646+0000", 2024-04-26T01:24:39.716 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597649+0000", 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1799999999999999e-06 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597677+0000", 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8368999999999999e-05 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.717 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597716+0000", 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8655999999999997e-05 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.597754+0000", 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8046000000000001e-05 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.698034+0000", 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100280385 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.718 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.698048+0000", 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4076999999999999e-05 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.798604+0000", 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10055604999999999 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.798691+0000", 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7213999999999995e-05 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.719 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/44 ECSubWrite(tid=71, reqid=client.4589.0:315, at_version=0'0, trim_to=0'0, roll_forward_to=46'20))", 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 21.441117550000001, 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30303612499999999, 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.720 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255436+0000", 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.81e-06 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.721 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255469+0000", 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2895999999999998e-05 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255499+0000", 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0253e-05 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.355601+0000", 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100101499 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.722 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455871+0000", 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10027049 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455899+0000", 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7636000000000001e-05 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558401+0000", 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102502418 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.723 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558470+0000", 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9122999999999995e-05 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.bs2 47/34 ECSubWrite(tid=127, reqid=client.4589.0:331, at_version=0'0, trim_to=0'0, roll_forward_to=47'8))", 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.559137+0000", 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 21.137414519, 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201584918, 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.724 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559137+0000", 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559137+0000", 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559139+0000", 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.584e-06 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.725 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559183+0000", 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3764999999999999e-05 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559207+0000", 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3915999999999999e-05 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.559222+0000", 2024-04-26T01:24:39.726 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5866999999999999e-05 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.759571+0000", 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200348216 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.759586+0000", 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5255999999999999e-05 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.727 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.760642+0000", 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010563320000000001 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.760722+0000", 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9981999999999994e-05 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=68, reqid=client.4589.0:383, at_version=51'42, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:39.728 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.378003246999999, 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.302538633, 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.729 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318551+0000", 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0930000000000001e-06 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318918+0000", 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036699299999999997 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.730 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318977+0000", 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9203999999999998e-05 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419092+0000", 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100115619 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419200+0000", 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010721300000000001 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.731 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419220+0000", 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0363000000000001e-05 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620972+0000", 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201752242 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621087+0000", 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114906 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=206, reqid=client.4589.0:388, at_version=51'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.354470908, 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.300923198, 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:24:39.733 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342083+0000", 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.793e-06 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.734 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342388+0000", 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030580700000000001 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342408+0000", 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9683000000000001e-05 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342417+0000", 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2550000000000003e-06 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.735 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.442698+0000", 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100280522 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.442716+0000", 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7808999999999998e-05 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.642922+0000", 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20020643499999999 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.736 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.643004+0000", 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1893999999999999e-05 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=69, reqid=client.4589.0:384, at_version=51'43, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.277415833999999, 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20207976799999999, 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.737 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.738 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419137+0000", 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.54e-07 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419193+0000", 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6375999999999997e-05 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419206+0000", 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.323e-05 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.739 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419217+0000", 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0906e-05 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519979+0000", 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10076136199999999 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519989+0000", 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0507e-05 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.740 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621131+0000", 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10114197 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621216+0000", 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4662999999999999e-05 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=70, reqid=client.4589.0:385, at_version=51'44, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:39.741 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.277315624, 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20264654900000001, 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:24:39.742 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419237+0000", 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8400000000000005e-07 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419295+0000", 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8211999999999998e-05 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419307+0000", 2024-04-26T01:24:39.743 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2558000000000001e-05 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519470+0000", 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100162563 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620565+0000", 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10109543 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620584+0000", 2024-04-26T01:24:39.744 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8352e-05 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621809+0000", 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012251160000000001 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621883+0000", 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3833999999999998e-05 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.745 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=207, reqid=client.4589.0:389, at_version=51'19, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.255132818, 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202618775, 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.746 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441420+0000", 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1489999999999999e-06 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441473+0000", 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2834999999999998e-05 2024-04-26T01:24:39.747 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441492+0000", 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9145000000000001e-05 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441503+0000", 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1151e-05 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.641827+0000", 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200323321 2024-04-26T01:24:39.748 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.641854+0000", 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7695000000000002e-05 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.643949+0000", 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020951979999999999 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.749 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.644038+0000", 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8281000000000004e-05 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=209, reqid=client.4589.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=51'20))", 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.848289481, 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.378972751, 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.750 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848264+0000", 2024-04-26T01:24:39.751 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3510000000000001e-06 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848286+0000", 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2011999999999999e-05 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848307+0000", 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1798999999999999e-05 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848322+0000", 2024-04-26T01:24:39.752 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4968e-05 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848406+0000", 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3762000000000002e-05 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848418+0000", 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1626e-05 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.227182+0000", 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.37876395099999999 2024-04-26T01:24:39.753 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.227235+0000", 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3282000000000002e-05 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 56/44 ECSubWrite(tid=107, reqid=osd.14.0:107, at_version=56'36, trim_to=0'0, roll_forward_to=55'33))", 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:28.148898+0000", 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.547653775000001, 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20199924699999999, 2024-04-26T01:24:39.754 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148898+0000", 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148898+0000", 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.755 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148899+0000", 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1699999999999998e-07 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148934+0000", 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5234999999999997e-05 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148989+0000", 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5135e-05 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.756 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.249123+0000", 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100133705 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.349389+0000", 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.1002661 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.349404+0000", 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5345999999999998e-05 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.757 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.350807+0000", 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014034119999999999 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.350897+0000", 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9796999999999994e-05 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.758 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 61/44 ECSubWrite(tid=127, reqid=client.4589.0:471, at_version=61'56, trim_to=0'0, roll_forward_to=59'47))", 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.971787+0000", 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.724764467, 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20188065899999999, 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971787+0000", 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.759 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971787+0000", 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971789+0000", 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.606e-06 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971822+0000", 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2762999999999998e-05 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.760 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971849+0000", 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7503000000000001e-05 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.071951+0000", 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10010151 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.172207+0000", 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10025627500000001 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.172233+0000", 2024-04-26T01:24:39.761 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6004999999999999e-05 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.173576+0000", 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001342789 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.173668+0000", 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2207999999999995e-05 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.762 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/44 ECSubWrite(tid=146, reqid=osd.14.0:146, at_version=67'60, trim_to=0'0, roll_forward_to=61'58))", 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.90622946500000001, 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20292310799999999, 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.763 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790323+0000", 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1299999999999999e-07 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790372+0000", 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8862000000000001e-05 2024-04-26T01:24:39.764 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790402+0000", 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9595999999999999e-05 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790435+0000", 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3872999999999999e-05 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.891521+0000", 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10108587600000001 2024-04-26T01:24:39.765 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.891538+0000", 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6816999999999998e-05 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.993158+0000", 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101619426 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.993245+0000", 2024-04-26T01:24:39.766 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7844999999999997e-05 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s2 67/44 ECSubRead(tid=111, to_read={3:1fe6984d:::smithi18128752-48:head=0,811008,0}, subchunks={3:1fe6984d:::smithi18128752-48:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.89925577800000001, 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.296535041, 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.767 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797298+0000", 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.66e-06 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.768 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797315+0000", 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7555999999999999e-05 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797346+0000", 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1192999999999998e-05 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797375+0000", 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8484999999999999e-05 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.769 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.093325+0000", 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29594971799999997 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.093831+0000", 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050642899999999997 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s2 67/63 ECSubWrite(tid=51, reqid=osd.0.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=67'33))", 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.234680+0000", 2024-04-26T01:24:39.770 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.46187169300000003, 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20149603299999999, 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234680+0000", 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234680+0000", 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.771 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234681+0000", 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.097e-06 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234705+0000", 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3405e-05 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.234733+0000", 2024-04-26T01:24:39.772 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8663999999999999e-05 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.334874+0000", 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100140251 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.435082+0000", 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100208967 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.435098+0000", 2024-04-26T01:24:39.773 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5906999999999999e-05 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.436140+0000", 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010415999999999999 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.436176+0000", 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6142000000000001e-05 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.774 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/44 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=67'64, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.361533936, 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20216706400000001, 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.775 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3700000000000001e-07 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.776 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335038+0000", 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0047e-05 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335047+0000", 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7329999999999992e-06 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.435145+0000", 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100097939 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.777 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.535402+0000", 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10025746200000001 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.535419+0000", 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6529e-05 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.537144+0000", 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017251930000000001 2024-04-26T01:24:39.778 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.537185+0000", 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0824000000000001e-05 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.779 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:39.780 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:217: finishing write tid 1 to smithi18128752-37 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:217: finishing write tid 2 to smithi18128752-37 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:217: finishing write tid 3 to smithi18128752-37 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:217: finishing write tid 4 to smithi18128752-37 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:213: done (6 left) 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:214: done (5 left) 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:215: done (4 left) 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:216: done (3 left) 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:219: done (2 left) 2024-04-26T01:24:39.782 INFO:tasks.rados.rados.0.smithi181.stdout:220: delete oid 30 current snap is 17 2024-04-26T01:24:39.783 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:24:39.783 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:39.785+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:39.804 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 7] out_osds: [12, 2] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:24:39.805 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:24:39.805 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-26T01:24:39.805 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.918518 2024-04-26T01:24:39.805 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0.9185178181236207 2024-04-26T01:24:39.830 INFO:tasks.rados.rados.0.smithi181.stdout:218: finishing copy_from to smithi18128752-41 2024-04-26T01:24:39.830 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 41 v 70 (ObjNum 83 snap 15 seq_num 83) dirty exists 2024-04-26T01:24:39.883 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:39.887+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:39.961 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:39.961 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:39.961 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.961 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 68/44 ECSubWriteReply(tid=117, last_complete=67'82, committed=1, applied=1))", 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.906356+0000", 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.060236372000000003, 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.060276720999999998, 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.906356+0000", 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.962 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.906356+0000", 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.906358+0000", 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3659999999999999e-06 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.906368+0000", 2024-04-26T01:24:39.963 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8970000000000003e-06 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.906396+0000", 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8683e-05 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 1 2024-04-26T01:24:39.964 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:39.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:24:39.979 INFO:tasks.rados.rados.0.smithi181.stdout:217: finishing write tid 5 to smithi18128752-37 2024-04-26T01:24:39.979 INFO:tasks.rados.rados.0.smithi181.stdout:217: oid 37 updating version 0 to 21 2024-04-26T01:24:39.979 INFO:tasks.rados.rados.0.smithi181.stdout:217: oid 37 version 21 is already newer than 19 2024-04-26T01:24:39.979 INFO:tasks.rados.rados.0.smithi181.stdout:217: oid 37 version 21 is already newer than 20 2024-04-26T01:24:39.979 INFO:tasks.rados.rados.0.smithi181.stdout:217: oid 37 version 21 is already newer than 18 2024-04-26T01:24:39.979 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 21 (ObjNum 87 snap 16 seq_num 87) dirty exists 2024-04-26T01:24:39.979 INFO:tasks.rados.rados.0.smithi181.stdout:217: left oid 37 (ObjNum 87 snap 16 seq_num 87) 2024-04-26T01:24:39.982 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:39.983+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:40.023 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:40.083 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:40.087+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:40.087 INFO:tasks.rados.rados.0.smithi181.stdout:217: done (2 left) 2024-04-26T01:24:40.087 INFO:tasks.rados.rados.0.smithi181.stdout:218: done (1 left) 2024-04-26T01:24:40.087 INFO:tasks.rados.rados.0.smithi181.stdout:220: done (0 left) 2024-04-26T01:24:40.087 INFO:tasks.rados.rados.0.smithi181.stdout:221: delete oid 26 current snap is 17 2024-04-26T01:24:40.103 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:40.104 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:40.104 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:40.104 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:40.104 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:40.118 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:24:40.168 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:40.168 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:40.184 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:40.189+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:40.186 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:40.187 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:40.208 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:40.208 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:40.229 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:40.229 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:40.244 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:40.244 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:40.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:129 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.671656665, 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.91360868200000001, 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 129 2024-04-26T01:24:40.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575106+0000", 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.043e-06 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575111+0000", 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3370000000000001e-06 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575128+0000", 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7600000000000001e-05 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575138+0000", 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.877914+0000", 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30277600900000001 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.878073+0000", 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015861299999999999 2024-04-26T01:24:40.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979098+0000", 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101025446 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979375+0000", 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000277309 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488714+0000", 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50933870599999997 2024-04-26T01:24:40.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:142 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.137120660000001, 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.88158457700000004, 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.250 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 142 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109644+0000", 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0970000000000002e-06 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110339+0000", 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00069456900000000002 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110373+0000", 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.358e-05 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110388+0000", 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5726000000000002e-05 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181183+0000", 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.070795134999999995 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181419+0000", 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023538399999999999 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.182735+0000", 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013160540000000001 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589098+0000", 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40636337 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991098+0000", 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401999885 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991226+0000", 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012777700000000001 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:143 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.130952997000001, 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87574223200000001, 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.255 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 143 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115811+0000", 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.317e-06 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116338+0000", 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052673300000000004 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116375+0000", 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6922999999999999e-05 2024-04-26T01:24:40.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116392+0000", 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6629999999999998e-05 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284095+0000", 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16770365000000001 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284265+0000", 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016969199999999999 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285568+0000", 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001302554 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589101+0000", 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30353306000000002 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:40.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991441+0000", 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40234051300000001 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991551+0000", 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011016 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:144 3.8s0 3:1fe6984d:::smithi18128752-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.130342601999999, 2024-04-26T01:24:40.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87544766600000001, 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 144 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116421+0000", 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.15e-06 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116427+0000", 2024-04-26T01:24:40.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9070000000000001e-06 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116443+0000", 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-05 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116452+0000", 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0890000000000006e-06 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286741+0000", 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170288786 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286883+0000", 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142683 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.287610+0000", 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000727172 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589207+0000", 2024-04-26T01:24:40.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30159656099999999 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991768+0000", 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40256125399999998 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991867+0000", 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8814000000000006e-05 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:145 3.8s0 3:1fe6984d:::smithi18128752-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 34.130296496, 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98038499800000001, 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 145 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.266 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5600000000000004e-07 2024-04-26T01:24:40.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116470+0000", 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116484+0000", 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3837e-05 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.216615+0000", 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100131365 2024-04-26T01:24:40.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992549+0000", 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77593446499999996 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992720+0000", 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017066299999999999 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096851+0000", 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104130502 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.269 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.958332095000003, 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60172694400000004, 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288438+0000", 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1620000000000008e-06 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288450+0000", 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1922999999999999e-05 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288486+0000", 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5905999999999998e-05 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388601+0000", 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10011487600000001 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890125+0000", 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50152462099999995 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890157+0000", 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1455999999999998e-05 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.273 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=17, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.858114196000002, 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50193089099999999, 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388648+0000", 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388649+0000", 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0270000000000001e-06 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388657+0000", 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0260000000000007e-06 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388677+0000", 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0069e-05 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488777+0000", 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100100183 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890549+0000", 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40177214999999999 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890579+0000", 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9436000000000001e-05 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.757861630999997, 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50191135200000003, 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.277 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488900+0000", 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488902+0000", 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9369999999999998e-06 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488913+0000", 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0789e-05 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488954+0000", 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0416000000000001e-05 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589090+0000", 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100136138 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990779+0000", 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401689352 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.990812+0000", 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2719999999999998e-05 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:160 3.8s0 3:1945891d:::smithi18128752-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.742146495, 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.79222307800000002, 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 160 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504618+0000", 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5600000000000001e-06 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504629+0000", 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0956e-05 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.283 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504661+0000", 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2428999999999997e-05 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504678+0000", 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6422999999999998e-05 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093363+0000", 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58868559200000004 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093514+0000", 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015052300000000001 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.094369+0000", 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00085476900000000001 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096327+0000", 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001957781 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296743+0000", 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20041606200000001 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296839+0000", 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5983000000000006e-05 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.288 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:269 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 28.073040331000001, 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81822368899999998, 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.289 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 269 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173723+0000", 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4399999999999999e-07 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173738+0000", 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5798999999999999e-05 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173754+0000", 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5943e-05 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173763+0000", 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1730000000000001e-06 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385572+0000", 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.211808092 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385724+0000", 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015250999999999999 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.292 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589244+0000", 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203520116 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589408+0000", 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016355699999999999 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.991945+0000", 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40253765499999999 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.293 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:383 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~720896] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.026014788000001, 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50440791100000004, 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 383 2024-04-26T01:24:40.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220749+0000", 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.995e-06 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221593+0000", 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00084381899999999999 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221614+0000", 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1061999999999999e-05 2024-04-26T01:24:40.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221623+0000", 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3130000000000004e-06 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.317149+0000", 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095525231000000002 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.317292+0000", 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014299199999999999 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318544+0000", 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001252567 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322759+0000", 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042146029999999999 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.724936+0000", 2024-04-26T01:24:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40217761699999999 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725155+0000", 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000218712 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:384 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~49152] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.025118592999998, 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50390550300000003, 2024-04-26T01:24:40.299 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 384 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:40.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221644+0000", 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0200000000000001e-07 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221675+0000", 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0929999999999997e-05 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221683+0000", 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3559999999999993e-06 2024-04-26T01:24:40.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221689+0000", 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7370000000000003e-06 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.319416+0000", 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.097727075999999996 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.319496+0000", 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9579000000000005e-05 2024-04-26T01:24:40.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.320065+0000", 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00056965599999999998 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322760+0000", 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026947189999999999 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725421+0000", 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40266085000000001 2024-04-26T01:24:40.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725549+0000", 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012789799999999999 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:385 3.8s0 3:14c0c421:::smithi18128752-31:head [setxattr _header (56) in=7b] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.025063071999998, 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50445409900000004, 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.305 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 385 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:40.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.82e-07 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221702+0000", 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.779e-06 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221715+0000", 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2337000000000001e-05 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221721+0000", 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.207e-06 2024-04-26T01:24:40.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.321880+0000", 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100158994 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.321983+0000", 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103669 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322488+0000", 2024-04-26T01:24:40.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050421999999999999 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.423268+0000", 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10077992299999999 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725960+0000", 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30269248500000001 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.726153+0000", 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000193003 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:386 3.8s0 3:14c0c421:::smithi18128752-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.025033094000001, 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70800976199999999, 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:40.311 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 386 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221732+0000", 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1129999999999999e-06 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221738+0000", 2024-04-26T01:24:40.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4609999999999996e-06 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221744+0000", 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3539999999999996e-06 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727164+0000", 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50541931600000001 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727250+0000", 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6618000000000003e-05 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929739+0000", 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20248843399999999 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:394 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:40.315 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.006376587999998, 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.79330826399999999, 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 394 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240388+0000", 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3410000000000001e-06 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240398+0000", 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8570000000000007e-06 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240421+0000", 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3198999999999999e-05 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240451+0000", 2024-04-26T01:24:40.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.023e-05 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727765+0000", 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.487313618 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727862+0000", 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7324999999999994e-05 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.033694+0000", 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30583169399999999 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:395 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.006288349999998, 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.89472404000000005, 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.320 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 395 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4199999999999995e-07 2024-04-26T01:24:40.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240479+0000", 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8199999999999996e-06 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240489+0000", 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7089999999999997e-06 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240496+0000", 2024-04-26T01:24:40.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9920000000000001e-06 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829732+0000", 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58923568800000004 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829882+0000", 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015056100000000001 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.135198+0000", 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.305315528 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:396 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.006232009000001, 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99608940400000001, 2024-04-26T01:24:40.325 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 396 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.326 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240531+0000", 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0800000000000004e-07 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240536+0000", 2024-04-26T01:24:40.327 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2059999999999998e-06 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240547+0000", 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1663e-05 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240555+0000", 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1010000000000003e-06 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.328 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831852+0000", 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.59129766500000003 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832019+0000", 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166773 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.236619+0000", 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40460038799999998 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.329 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 60/44 ECSubReadReply(tid=93, attrs_read=0))", 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.690655+0000", 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 7.5561072710000001, 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50236763200000001, 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.330 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.690655+0000", 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.690655+0000", 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.690656+0000", 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8999999999999995e-07 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.331 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.690680+0000", 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4145999999999999e-05 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.690694+0000", 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4487999999999999e-05 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.790772+0000", 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100077902 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.192527+0000", 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40175459600000002 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.193022+0000", 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00049571000000000001 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.333 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 60/44 ECSubWriteReply(tid=92, last_complete=60'59, committed=1, applied=1))", 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.790860+0000", 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 7.4559023809999996, 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50264487400000002, 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.790860+0000", 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.790860+0000", 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.790860+0000", 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1800000000000001e-07 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.790864+0000", 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5839999999999999e-06 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.790874+0000", 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0242e-05 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.790884+0000", 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6670000000000008e-06 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.293471+0000", 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50258754800000005 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.293505+0000", 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3414999999999998e-05 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 60/44 ECSubWriteReply(tid=92, last_complete=60'59, committed=1, applied=1))", 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.791027+0000", 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 7.4557347170000003, 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50300713100000005, 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:40.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.791027+0000", 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.791027+0000", 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.791028+0000", 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7e-07 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.791031+0000", 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8990000000000001e-06 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.791040+0000", 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8660000000000001e-06 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.791050+0000", 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1249999999999999e-06 2024-04-26T01:24:40.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.293897+0000", 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50284751000000005 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.294034+0000", 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013736099999999999 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:40.340 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:40.342 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:40.342 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:40.343 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:24:40.343 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:40.345+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:40.354 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:40.354 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:40.383 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:40.383 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:40.393 INFO:tasks.rados.rados.0.smithi181.stdout:221: done (0 left) 2024-04-26T01:24:40.393 INFO:tasks.rados.rados.0.smithi181.stdout:222: copy_from oid 20 from oid 42 current snap is 17 2024-04-26T01:24:40.393 INFO:tasks.rados.rados.0.smithi181.stdout:223: delete oid 5 current snap is 17 2024-04-26T01:24:40.416 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:40.417 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:40.442 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:40.445+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:40.446 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:40.446 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:40.473 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:40.473 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:40.474 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:40.474 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:40.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:40.501 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:40.501 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:40.506 INFO:tasks.rados.rados.0.smithi181.stdout:223: done (1 left) 2024-04-26T01:24:40.506 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 13 current snap is 17 2024-04-26T01:24:40.506 INFO:tasks.rados.rados.0.smithi181.stdout:224: seq_num 91 ranges {1474560=770048,2244608=475136,2719744=294912} 2024-04-26T01:24:40.533 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:40.534 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:40.543 INFO:tasks.rados.rados.0.smithi181.stdout:224: writing smithi18128752-13 from 1474560 to 2244608 tid 1 2024-04-26T01:24:40.543 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:40.547+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:40.552 INFO:tasks.rados.rados.0.smithi181.stdout:224: writing smithi18128752-13 from 2244608 to 2719744 tid 2 2024-04-26T01:24:40.557 INFO:tasks.rados.rados.0.smithi181.stdout:224: writing smithi18128752-13 from 2719744 to 3014656 tid 3 2024-04-26T01:24:40.557 INFO:tasks.rados.rados.0.smithi181.stdout:225: read oid 19 snap -1 2024-04-26T01:24:40.557 INFO:tasks.rados.rados.0.smithi181.stdout:225: expect deleted 2024-04-26T01:24:40.557 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 21 current snap is 17 2024-04-26T01:24:40.557 INFO:tasks.rados.rados.0.smithi181.stdout:226: seq_num 92 ranges {2588672=786432,3375104=491520,3866624=278528} 2024-04-26T01:24:40.562 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:40.562 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:40.589 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:40.590 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:40.591 INFO:tasks.rados.rados.0.smithi181.stdout:226: writing smithi18128752-21 from 2588672 to 3375104 tid 1 2024-04-26T01:24:40.595 INFO:tasks.rados.rados.0.smithi181.stdout:226: writing smithi18128752-21 from 3375104 to 3866624 tid 2 2024-04-26T01:24:40.597 INFO:tasks.rados.rados.0.smithi181.stdout:226: writing smithi18128752-21 from 3866624 to 4145152 tid 3 2024-04-26T01:24:40.597 INFO:tasks.rados.rados.0.smithi181.stdout:227: delete oid 22 current snap is 17 2024-04-26T01:24:40.608 INFO:tasks.rados.rados.0.smithi181.stdout:222: finishing copy_from racing read to smithi18128752-20 2024-04-26T01:24:40.620 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:40.621 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:40.635 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:40.682 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:40.682 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:40.686 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:40.691+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:40.717 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:40.717 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:40.745 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:40.749+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:40.759 INFO:tasks.rados.rados.0.smithi181.stdout:224: finishing write tid 1 to smithi18128752-13 2024-04-26T01:24:40.762 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:40.762 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:40.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:227: done (4 left) 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:228: read oid 23 snap -1 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:228: expect deleted 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:229: copy_from oid 24 from oid 48 current snap is 17 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:230: copy_from oid 37 from oid 10 current snap is 17 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:231: read oid 25 snap 12 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:231: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-26T01:24:40.803 INFO:tasks.rados.rados.0.smithi181.stdout:232: delete oid 47 current snap is 17 2024-04-26T01:24:40.851 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.5 primary-affinity to 0.918518 (8601952) 2024-04-26T01:24:40.866 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:40.871+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:40.903 INFO:tasks.rados.rados.0.smithi181.stdout:222: finishing copy_from to smithi18128752-20 2024-04-26T01:24:40.904 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 14 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-26T01:24:40.945 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:40.947+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:40.947 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:40.947 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:40.961 INFO:tasks.rados.rados.0.smithi181.stdout:224: finishing write tid 2 to smithi18128752-13 2024-04-26T01:24:40.962 INFO:tasks.rados.rados.0.smithi181.stdout:224: finishing write tid 3 to smithi18128752-13 2024-04-26T01:24:40.962 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:40.971 INFO:tasks.rados.rados.0.smithi181.stdout:230: finishing copy_from racing read to smithi18128752-37 2024-04-26T01:24:41.004 INFO:tasks.rados.rados.0.smithi181.stdout:226: finishing write tid 1 to smithi18128752-21 2024-04-26T01:24:41.004 INFO:tasks.rados.rados.0.smithi181.stdout:226: finishing write tid 2 to smithi18128752-21 2024-04-26T01:24:41.009 INFO:tasks.rados.rados.0.smithi181.stdout:222: done (8 left) 2024-04-26T01:24:41.009 INFO:tasks.rados.rados.0.smithi181.stdout:225: done (7 left) 2024-04-26T01:24:41.009 INFO:tasks.rados.rados.0.smithi181.stdout:228: done (6 left) 2024-04-26T01:24:41.009 INFO:tasks.rados.rados.0.smithi181.stdout:232: done (5 left) 2024-04-26T01:24:41.009 INFO:tasks.rados.rados.0.smithi181.stdout:233: rollback oid 20 current snap is 17 2024-04-26T01:24:41.009 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 20 to 16 2024-04-26T01:24:41.010 INFO:tasks.rados.rados.0.smithi181.stdout:234: delete oid 2 current snap is 17 2024-04-26T01:24:41.046 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:41.047+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:41.080 INFO:tasks.rados.rados.0.smithi181.stdout:229: finishing copy_from racing read to smithi18128752-24 2024-04-26T01:24:41.111 INFO:tasks.rados.rados.0.smithi181.stdout:230: finishing copy_from to smithi18128752-37 2024-04-26T01:24:41.111 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 11 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-04-26T01:24:41.133 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:41.133 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:41.147 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:41.148 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:41.153+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:41.185 INFO:tasks.rados.rados.0.smithi181.stdout:231: expect (ObjNum 29 snap 0 seq_num 29) 2024-04-26T01:24:41.223 INFO:tasks.rados.rados.0.smithi181.stdout:226: finishing write tid 3 to smithi18128752-21 2024-04-26T01:24:41.223 INFO:tasks.rados.rados.0.smithi181.stdout:226: finishing write tid 4 to smithi18128752-21 2024-04-26T01:24:41.223 INFO:tasks.rados.rados.0.smithi181.stdout:229: finishing copy_from to smithi18128752-24 2024-04-26T01:24:41.223 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 59 (ObjNum 79 snap 12 seq_num 79) dirty exists 2024-04-26T01:24:41.237 INFO:tasks.rados.rados.0.smithi181.stdout:224: finishing write tid 4 to smithi18128752-13 2024-04-26T01:24:41.248 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:41.251+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:41.276 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:41.276 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:41.290 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:41.314 INFO:tasks.rados.rados.0.smithi181.stdout:233: finishing rollback tid 0 to smithi18128752-20 2024-04-26T01:24:41.349 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:41.351+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:41.418 INFO:tasks.rados.rados.0.smithi181.stdout:229: done (6 left) 2024-04-26T01:24:41.418 INFO:tasks.rados.rados.0.smithi181.stdout:230: done (5 left) 2024-04-26T01:24:41.418 INFO:tasks.rados.rados.0.smithi181.stdout:231: done (4 left) 2024-04-26T01:24:41.418 INFO:tasks.rados.rados.0.smithi181.stdout:234: done (3 left) 2024-04-26T01:24:41.418 INFO:tasks.rados.rados.0.smithi181.stdout:235: copy_from oid 41 from oid 12 current snap is 17 2024-04-26T01:24:41.418 INFO:tasks.rados.rados.0.smithi181.stdout:236: rollback oid 45 current snap is 17 2024-04-26T01:24:41.418 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 45 to 16 2024-04-26T01:24:41.419 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 17 current snap is 17 2024-04-26T01:24:41.419 INFO:tasks.rados.rados.0.smithi181.stdout:237: seq_num 96 ranges {1294336=458752,1753088=688128,2441216=475136} 2024-04-26T01:24:41.423 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:41.423 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:41.435 INFO:tasks.rados.rados.0.smithi181.stdout:237: writing smithi18128752-17 from 1294336 to 1753088 tid 1 2024-04-26T01:24:41.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:41.442 INFO:tasks.rados.rados.0.smithi181.stdout:237: writing smithi18128752-17 from 1753088 to 2441216 tid 2 2024-04-26T01:24:41.446 INFO:tasks.rados.rados.0.smithi181.stdout:237: writing smithi18128752-17 from 2441216 to 2916352 tid 3 2024-04-26T01:24:41.446 INFO:tasks.rados.rados.0.smithi181.stdout:238: rollback oid 10 current snap is 17 2024-04-26T01:24:41.447 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 10 to 16 2024-04-26T01:24:41.447 INFO:tasks.rados.rados.0.smithi181.stdout:239: snap_remove snap 14 2024-04-26T01:24:41.450 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:41.455+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:41.551 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:41.555+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:41.606 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:41.607 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:41.622 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:41.652 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:41.655+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:41.746 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:41.746 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:41.753 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:41.755+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:41.760 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:41.796 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:24:41.853 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:41.857+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:41.875 INFO:tasks.rados.rados.0.smithi181.stdout:239: done (7 left) 2024-04-26T01:24:41.875 INFO:tasks.rados.rados.0.smithi181.stdout:240: snap_remove snap 12 2024-04-26T01:24:41.885 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:41.885 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:41.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:41.954 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:41.959+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:42.018 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:42.019 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:42.035 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:42.056 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:42.059+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:42.156 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:42.159+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:42.160 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:42.161 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:42.175 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:42.258 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:42.263+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:42.298 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:42.299 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:42.312 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:42.359 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:42.363+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:42.438 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:42.438 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:42.453 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:42.460 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:42.463+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:42.560 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:42.565+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:42.574 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:42.574 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:42.589 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:42.661 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:42.663+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:42.712 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:42.713 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:42.727 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:42.744 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:42.772 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:42.775+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:42.850 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:42.850 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:42.863 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:42.863+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:42.864 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:42.878 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:24:42.884 INFO:tasks.rados.rados.0.smithi181.stdout:240: done (7 left) 2024-04-26T01:24:42.884 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 47 current snap is 17 2024-04-26T01:24:42.884 INFO:tasks.rados.rados.0.smithi181.stdout:241: seq_num 97 ranges {0=475136,475136=442368,917504=442368,1359872=278528} 2024-04-26T01:24:42.891 INFO:tasks.rados.rados.0.smithi181.stdout:241: writing smithi18128752-47 from 0 to 475136 tid 1 2024-04-26T01:24:42.898 INFO:tasks.rados.rados.0.smithi181.stdout:241: writing smithi18128752-47 from 475136 to 917504 tid 2 2024-04-26T01:24:42.906 INFO:tasks.rados.rados.0.smithi181.stdout:241: writing smithi18128752-47 from 917504 to 1359872 tid 3 2024-04-26T01:24:42.911 INFO:tasks.rados.rados.0.smithi181.stdout:241: writing smithi18128752-47 from 1359872 to 1638400 tid 4 2024-04-26T01:24:42.911 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:42.912 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:242: copy_from oid 48 from oid 42 current snap is 17 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:224: finishing write tid 5 to smithi18128752-13 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:224: oid 13 updating version 0 to 75 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:224: oid 13 version 75 is already newer than 72 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:224: oid 13 version 75 is already newer than 74 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:224: oid 13 version 75 is already newer than 73 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 75 (ObjNum 91 snap 17 seq_num 91) dirty exists 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:224: left oid 13 (ObjNum 91 snap 17 seq_num 91) 2024-04-26T01:24:42.913 INFO:tasks.rados.rados.0.smithi181.stdout:226: finishing write tid 5 to smithi18128752-21 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:226: oid 21 updating version 0 to 58 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:226: oid 21 version 58 is already newer than 56 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:226: oid 21 version 58 is already newer than 57 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:226: oid 21 updating version 58 to 59 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 21 v 59 (ObjNum 92 snap 17 seq_num 92) dirty exists 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:226: left oid 21 (ObjNum 92 snap 17 seq_num 92) 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:233: finishing rollback tid 1 to smithi18128752-20 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:233: finishing rollback tid 2 to smithi18128752-20 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 22 (ObjNum 81 snap 13 seq_num 81) dirty exists 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing rollback tid 0 to smithi18128752-10 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:224: done (8 left) 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:226: done (7 left) 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:233: done (6 left) 2024-04-26T01:24:42.914 INFO:tasks.rados.rados.0.smithi181.stdout:243: delete oid 37 current snap is 17 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:235: finishing copy_from racing read to smithi18128752-41 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:237: finishing write tid 1 to smithi18128752-17 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:236: finishing rollback tid 0 to smithi18128752-45 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:237: finishing write tid 2 to smithi18128752-17 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:237: finishing write tid 3 to smithi18128752-17 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing rollback tid 1 to smithi18128752-10 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:235: finishing copy_from to smithi18128752-41 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 41 v 19 (ObjNum 85 snap 16 seq_num 85) dirty exists 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:237: finishing write tid 4 to smithi18128752-17 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:236: finishing rollback tid 1 to smithi18128752-45 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:236: finishing rollback tid 2 to smithi18128752-45 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 79 (ObjNum 80 snap 13 seq_num 80) dirty exists 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing rollback tid 2 to smithi18128752-10 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 41 (ObjNum 53 snap 1 seq_num 53) dirty exists 2024-04-26T01:24:42.915 INFO:tasks.rados.rados.0.smithi181.stdout:237: finishing write tid 5 to smithi18128752-17 2024-04-26T01:24:42.916 INFO:tasks.rados.rados.0.smithi181.stdout:237: oid 17 updating version 0 to 62 2024-04-26T01:24:42.916 INFO:tasks.rados.rados.0.smithi181.stdout:237: oid 17 updating version 62 to 63 2024-04-26T01:24:42.916 INFO:tasks.rados.rados.0.smithi181.stdout:237: oid 17 version 63 is already newer than 61 2024-04-26T01:24:42.916 INFO:tasks.rados.rados.0.smithi181.stdout:237: oid 17 updating version 63 to 64 2024-04-26T01:24:42.916 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 64 (ObjNum 96 snap 17 seq_num 96) dirty exists 2024-04-26T01:24:42.916 INFO:tasks.rados.rados.0.smithi181.stdout:237: left oid 17 (ObjNum 96 snap 17 seq_num 96) 2024-04-26T01:24:42.963 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:42.963+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:43.028 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:43.028 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:43.032 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:43.033 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:43.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:43.085 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:43.087+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:43.154 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:43.154 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:43.166 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:43.167+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:43.176 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:43.176 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:43.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:43.201 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:43.201 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:43.267 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:43.271+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:43.312 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:43.312 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:43.312 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:43.312 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:43.313 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:43.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:43.352 INFO:tasks.rados.rados.0.smithi181.stdout:235: done (6 left) 2024-04-26T01:24:43.352 INFO:tasks.rados.rados.0.smithi181.stdout:236: done (5 left) 2024-04-26T01:24:43.352 INFO:tasks.rados.rados.0.smithi181.stdout:237: done (4 left) 2024-04-26T01:24:43.352 INFO:tasks.rados.rados.0.smithi181.stdout:238: done (3 left) 2024-04-26T01:24:43.352 INFO:tasks.rados.rados.0.smithi181.stdout:243: done (2 left) 2024-04-26T01:24:43.352 INFO:tasks.rados.rados.0.smithi181.stdout:244: delete oid 39 current snap is 17 2024-04-26T01:24:43.368 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:43.367+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:43.403 INFO:tasks.rados.rados.0.smithi181.stdout:241: finishing write tid 1 to smithi18128752-47 2024-04-26T01:24:43.404 INFO:tasks.rados.rados.0.smithi181.stdout:241: finishing write tid 2 to smithi18128752-47 2024-04-26T01:24:43.453 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:43.453 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:43.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:43.453 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:169 3.as0 3:56e8b2fd:::smithi18128752-12:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.158551629000002, 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.415445014, 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 169 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.454 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297980+0000", 2024-04-26T01:24:43.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3359999999999997e-06 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297998+0000", 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8499999999999999e-05 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298041+0000", 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3171999999999999e-05 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398202+0000", 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10016098800000001 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398693+0000", 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049036800000000003 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398913+0000", 2024-04-26T01:24:43.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220051 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.610194+0000", 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.211280998 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612042+0000", 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018484860000000001 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713242+0000", 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10119953 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713420+0000", 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017858500000000001 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.459 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:170 3.as0 3:56e8b2fd:::smithi18128752-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.058286846999998, 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.213653854, 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 170 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398242+0000", 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0530000000000001e-06 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398252+0000", 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4830000000000005e-06 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398271+0000", 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-05 2024-04-26T01:24:43.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398282+0000", 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1476e-05 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404421+0000", 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006138592 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404504+0000", 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2750999999999994e-05 2024-04-26T01:24:43.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611404+0000", 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20690029200000001 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611712+0000", 2024-04-26T01:24:43.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030829799999999998 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611894+0000", 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018167899999999999 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:212 3.as0 3:56e8b2fd:::smithi18128752-12:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 35.335528197999999, 2024-04-26T01:24:43.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.306234695, 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121002+0000", 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.366e-06 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121071+0000", 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9202000000000006e-05 2024-04-26T01:24:43.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121106+0000", 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5052999999999998e-05 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121125+0000", 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8219e-05 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.468 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221415+0000", 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029012800000001 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221427+0000", 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.187e-05 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222099+0000", 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067277899999999998 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222322+0000", 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000222272 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:43.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.224059+0000", 2024-04-26T01:24:43.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017374059999999999 2024-04-26T01:24:43.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.472 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226116+0000", 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020565679999999999 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426988+0000", 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20087223500000001 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.427234+0000", 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024559700000000001 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.473 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 32.230773235000001, 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28191508700000001, 2024-04-26T01:24:43.474 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225756+0000", 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.756e-06 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225765+0000", 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225784+0000", 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.325876+0000", 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100092051 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507611+0000", 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.181735758 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507669+0000", 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7361999999999997e-05 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=10, last_complete=40'9, committed=1, applied=1))", 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 32.149470489000002, 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044763199999999, 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.479 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307057+0000", 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307065+0000", 2024-04-26T01:24:43.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9039999999999992e-06 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307072+0000", 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5619999999999998e-06 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.307078+0000", 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1830000000000002e-06 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507467+0000", 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200389598 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507504+0000", 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6879999999999999e-05 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:43.482 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.053710685999999, 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20097827700000001, 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402817+0000", 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.026e-06 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402832+0000", 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4684999999999999e-05 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402851+0000", 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9338e-05 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402863+0000", 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1832e-05 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.503005+0000", 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014197399999999 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.503027+0000", 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1903999999999999e-05 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.603709+0000", 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10068225 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.603795+0000", 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5267999999999997e-05 2024-04-26T01:24:43.486 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 41/30 ECSubWrite(tid=94, reqid=osd.4.0:94, at_version=41'18, trim_to=0'0, roll_forward_to=39'14))", 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.952464656, 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202154895, 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504063+0000", 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4900000000000001e-07 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504085+0000", 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2342999999999999e-05 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504093+0000", 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504100+0000", 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2470000000000002e-06 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704414+0000", 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20031389399999999 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704439+0000", 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.499e-05 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706155+0000", 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017162480000000001 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.490 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706217+0000", 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1878999999999995e-05 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/30 ECSubWrite(tid=97, reqid=client.4589.0:400, at_version=52'20, trim_to=0'0, roll_forward_to=51'18))", 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.524595408, 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199335239, 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.491 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931932+0000", 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8000000000000001e-07 2024-04-26T01:24:43.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931970+0000", 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8256000000000001e-05 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.931998+0000", 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8186000000000001e-05 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.932022+0000", 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3269999999999999e-05 2024-04-26T01:24:43.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.129435+0000", 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19741354999999999 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.129451+0000", 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6195999999999999e-05 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.131180+0000", 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017288760000000001 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.131267+0000", 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6525000000000003e-05 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s2 52/44 ECSubWrite(tid=213, reqid=osd.13.0:212, at_version=0'0, trim_to=0'0, roll_forward_to=52'22))", 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 19.098856383000001, 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30091122399999998, 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:43.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357672+0000", 2024-04-26T01:24:43.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.75e-06 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357708+0000", 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5781999999999997e-05 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357774+0000", 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6131999999999996e-05 2024-04-26T01:24:43.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.457877+0000", 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100102935 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458051+0000", 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017357000000000001 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458066+0000", 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5526e-05 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658511+0000", 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044448300000001 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658582+0000", 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1045999999999997e-05 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.501 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s2 61/44 ECSubWrite(tid=246, reqid=osd.13.0:243, at_version=0'0, trim_to=0'0, roll_forward_to=61'39))", 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:33.530442+0000", 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.9260850450000007, 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200375104, 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.530442+0000", 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.530442+0000", 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.530443+0000", 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.283e-06 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.530462+0000", 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.891e-05 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.530483+0000", 2024-04-26T01:24:43.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0463e-05 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.530498+0000", 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5475999999999999e-05 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.630627+0000", 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10012873899999999 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.630641+0000", 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4242e-05 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.730746+0000", 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100104467 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.730817+0000", 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1524000000000002e-05 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.845853+0000", 2024-04-26T01:24:43.507 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6106739000000001, 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199408529, 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.845853+0000", 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.845853+0000", 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.845855+0000", 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.733e-06 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.845865+0000", 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0103e-05 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.845898+0000", 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2889000000000003e-05 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.845919+0000", 2024-04-26T01:24:43.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1166999999999999e-05 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045194+0000", 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19927509500000001 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045262+0000", 2024-04-26T01:24:43.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7541999999999998e-05 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.846027+0000", 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6104998310000003, 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19937539300000001, 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.512 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846027+0000", 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846027+0000", 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846029+0000", 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5099999999999999e-06 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846038+0000", 2024-04-26T01:24:43.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0929999999999992e-06 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846052+0000", 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4636e-05 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.515 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846063+0000", 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0866999999999999e-05 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045349+0000", 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19928612800000001 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045403+0000", 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3158999999999997e-05 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.519 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:24:43.519 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.846142+0000", 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.610384797, 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19938701, 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846142+0000", 2024-04-26T01:24:43.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.521 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.521 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846142+0000", 2024-04-26T01:24:43.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846143+0000", 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8599999999999997e-07 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846148+0000", 2024-04-26T01:24:43.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2769999999999996e-06 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846165+0000", 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6555999999999999e-05 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846175+0000", 2024-04-26T01:24:43.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0214000000000001e-05 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045494+0000", 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19931837099999999 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045529+0000", 2024-04-26T01:24:43.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5706e-05 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.846149+0000", 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6103776740000004, 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199548432, 2024-04-26T01:24:43.525 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846149+0000", 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846149+0000", 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846151+0000", 2024-04-26T01:24:43.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.956e-06 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846160+0000", 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1269999999999992e-06 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846191+0000", 2024-04-26T01:24:43.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0821999999999997e-05 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846214+0000", 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2416999999999999e-05 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045628+0000", 2024-04-26T01:24:43.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19941462500000001 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045698+0000", 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9485000000000003e-05 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 65/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.846395+0000", 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6101320269999997, 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199456465, 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846395+0000", 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846395+0000", 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846396+0000", 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.327e-06 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846403+0000", 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0859999999999996e-06 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846422+0000", 2024-04-26T01:24:43.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8544000000000001e-05 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.846439+0000", 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6985000000000001e-05 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045802+0000", 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19936319699999999 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.045851+0000", 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9326000000000002e-05 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.535 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.as0 65/63 ECSubRead(tid=24, to_read={3:56e8b2fd:::smithi18128752-12:head=0,4096,0}, subchunks={3:56e8b2fd:::smithi18128752-12:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5117184720000001, 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20271442200000001, 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.536 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944842+0000", 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3609000000000003e-05 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944872+0000", 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9445e-05 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147126+0000", 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202254668 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147523+0000", 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039669999999999999 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.as0 65/63 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5107931350000001, 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20196552700000001, 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.541 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945736+0000", 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945793+0000", 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7462e-05 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945835+0000", 2024-04-26T01:24:43.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1705999999999997e-05 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945854+0000", 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9298999999999999e-05 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147627+0000", 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20177293299999999 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147699+0000", 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2384000000000004e-05 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 66/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5099501750000002, 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201296893, 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.545 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946579+0000", 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0360000000000001e-06 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946590+0000", 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1583000000000001e-05 2024-04-26T01:24:43.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946624+0000", 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3849000000000001e-05 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946648+0000", 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3326999999999999e-05 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147789+0000", 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201141864 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147874+0000", 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4233999999999993e-05 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.as0 66/63 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:24:43.549 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5097868989999998, 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30137683399999998, 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:24:43.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946741+0000", 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7599999999999997e-07 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946762+0000", 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0993e-05 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946792+0000", 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9776000000000001e-05 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946814+0000", 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2889999999999999e-05 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.248046+0000", 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30123198000000001 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.248117+0000", 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0518999999999993e-05 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 66/63 ECSubWriteReply(tid=26, last_complete=66'21, committed=1, applied=1))", 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.455416+0000", 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.001110948, 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200451711, 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455416+0000", 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455416+0000", 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455418+0000", 2024-04-26T01:24:43.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.697e-06 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455425+0000", 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5830000000000001e-06 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455454+0000", 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8889000000000001e-05 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.555551+0000", 2024-04-26T01:24:43.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009688 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.655817+0000", 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.1002654 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.655868+0000", 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1261999999999998e-05 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:43.556 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:43.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:43.557 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:43.521+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:43.558 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:43.558 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:43.565 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:43.565 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:43.620 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:43.625+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:43.646 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:43.646 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:43.692 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:43.692 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:43.692 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:43.692 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:43.693 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:43.693 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:43.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:43.718 INFO:tasks.rados.rados.0.smithi181.stdout:242: finishing copy_from racing read to smithi18128752-48 2024-04-26T01:24:43.720 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:43.723+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:43.729 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:43.729 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:43.756 INFO:tasks.rados.rados.0.smithi181.stdout:244: done (2 left) 2024-04-26T01:24:43.756 INFO:tasks.rados.rados.0.smithi181.stdout:245: setattr oid 8 current snap is 17 2024-04-26T01:24:43.757 INFO:tasks.rados.rados.0.smithi181.stdout:246: snap_remove snap 16 2024-04-26T01:24:43.765 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:43.766 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:43.776 INFO:tasks.rados.rados.0.smithi181.stdout:246: done (3 left) 2024-04-26T01:24:43.776 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 43 current snap is 17 2024-04-26T01:24:43.776 INFO:tasks.rados.rados.0.smithi181.stdout:247: seq_num 100 ranges {1703936=524288,2228224=753664,2981888=409600} 2024-04-26T01:24:43.790 INFO:tasks.rados.rados.0.smithi181.stdout:247: writing smithi18128752-43 from 1703936 to 2228224 tid 1 2024-04-26T01:24:43.796 INFO:tasks.rados.rados.0.smithi181.stdout:247: writing smithi18128752-43 from 2228224 to 2981888 tid 2 2024-04-26T01:24:43.799 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:43.799 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:43.799 INFO:tasks.rados.rados.0.smithi181.stdout:247: writing smithi18128752-43 from 2981888 to 3391488 tid 3 2024-04-26T01:24:43.800 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 28 current snap is 17 2024-04-26T01:24:43.800 INFO:tasks.rados.rados.0.smithi181.stdout:248: seq_num 101 ranges {458752=540672,999424=507904,1507328=622592,2129920=32768} 2024-04-26T01:24:43.806 INFO:tasks.rados.rados.0.smithi181.stdout:248: writing smithi18128752-28 from 458752 to 999424 tid 1 2024-04-26T01:24:43.810 INFO:tasks.rados.rados.0.smithi181.stdout:248: writing smithi18128752-28 from 999424 to 1507328 tid 2 2024-04-26T01:24:43.815 INFO:tasks.rados.rados.0.smithi181.stdout:248: writing smithi18128752-28 from 1507328 to 2129920 tid 3 2024-04-26T01:24:43.815 INFO:tasks.rados.rados.0.smithi181.stdout:248: writing smithi18128752-28 from 2129920 to 2162688 tid 4 2024-04-26T01:24:43.815 INFO:tasks.rados.rados.0.smithi181.stdout:249: copy_from oid 30 from oid 21 current snap is 17 2024-04-26T01:24:43.815 INFO:tasks.rados.rados.0.smithi181.stdout:241: finishing write tid 3 to smithi18128752-47 2024-04-26T01:24:43.815 INFO:tasks.rados.rados.0.smithi181.stdout:250: snap_create 2024-04-26T01:24:43.815 INFO:tasks.rados.rados.0.smithi181.stdout:241: finishing write tid 4 to smithi18128752-47 2024-04-26T01:24:43.822 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:43.825+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:43.850 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:43.850 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:43.850 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:43.850 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:43.850 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:43.865 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:24:43.901 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:43.901 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:43.910 INFO:tasks.rados.rados.0.smithi181.stdout:241: finishing write tid 5 to smithi18128752-47 2024-04-26T01:24:43.923 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:43.925+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:43.944 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:43.944 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:44.024 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:44.027+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPush(3.ds0 66/63 [PushOp(3:b136579b:::smithi18128752-36:4, version: 43'19, data_included: [0~147456], data_size: 147456, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:b136579b:::smithi18128752-36:4@43'19, size: 589824, copy_subset: [], clone_subset: {}, snapset: 7=[]:{4: [4]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 589824, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.048505+0000", 2024-04-26T01:24:44.037 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.9926768189999997, 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112790265, 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.048505+0000", 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.048505+0000", 2024-04-26T01:24:44.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.048505+0000", 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.048505+0000", 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.048548+0000", 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3012000000000001e-05 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.148748+0000", 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100200619 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.159146+0000", 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010397849000000001 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.161295+0000", 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021487849999999998 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGRecoveryDeleteReply(3.ds0 e66,63 3:b136579b:::smithi18128752-36:6,48'23)", 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.056963+0000", 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.9842182319999999, 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102065646, 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.041 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.056963+0000", 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.056963+0000", 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.056970+0000", 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1600000000000001e-06 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.056981+0000", 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1090000000000001e-05 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.057018+0000", 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7032000000000001e-05 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.057035+0000", 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.658e-05 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.157342+0000", 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100306665 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.159029+0000", 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016871189999999999 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.044 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 66/63 ECSubReadReply(tid=4, attrs_read=0))", 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.164720+0000", 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.8764618479999999, 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101855427, 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.164720+0000", 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.164720+0000", 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.164721+0000", 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.75e-06 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.165315+0000", 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00059402599999999999 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.165346+0000", 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0301999999999999e-05 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.265431+0000", 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100085641 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.265813+0000", 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000381624 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.266575+0000", 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000762084 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGPushReply(3.ds0 66/63 [PushReplyOp(3:b2e0ecb3:::smithi18128752-14:head)])", 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.268758+0000", 2024-04-26T01:24:44.048 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.7724238640000003, 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101009495, 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.268758+0000", 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.268758+0000", 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.268759+0000", 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8139999999999999e-06 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.268773+0000", 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3175e-05 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.268802+0000", 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9567e-05 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.268826+0000", 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3689e-05 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.369097+0000", 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10027117000000001 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.369767+0000", 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067007999999999998 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.as1 66/63 ECSubWrite(tid=26, reqid=osd.3.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=66'21))", 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.453722+0000", 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5874592590000001, 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201058036, 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.052 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453722+0000", 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453722+0000", 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453724+0000", 2024-04-26T01:24:44.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1100000000000001e-06 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453753+0000", 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8382999999999999e-05 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453805+0000", 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2707999999999997e-05 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453830+0000", 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4873999999999999e-05 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.554095+0000", 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10026523900000001 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.554111+0000", 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6025999999999999e-05 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.654721+0000", 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10060960100000001 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.654780+0000", 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9095000000000003e-05 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 66/63 ECSubReadReply(tid=5, attrs_read=0))", 2024-04-26T01:24:44.056 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.471340+0000", 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5698415780000001, 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100973229, 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.471340+0000", 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.471340+0000", 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.471342+0000", 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-06 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.471491+0000", 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014888299999999999 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.471519+0000", 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8010000000000001e-05 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.471533+0000", 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4069e-05 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.571813+0000", 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10028087300000001 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.572313+0000", 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049957400000000002 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 66/63 ECSubReadReply(tid=6, attrs_read=1))", 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.579534+0000", 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.4616473079999999, 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101140587, 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.060 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.579534+0000", 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.579534+0000", 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.579536+0000", 2024-04-26T01:24:44.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6700000000000001e-06 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.579652+0000", 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011595399999999999 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.579675+0000", 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2952e-05 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.579688+0000", 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3360000000000001e-05 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.680059+0000", 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100371086 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.680675+0000", 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061556499999999999 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:542 3.ds0 3:b6ed1d05:::smithi18128752-41:head [copy-from ver 70] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2018459320000003, 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99677863200000005, 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 542 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.064 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839338+0000", 2024-04-26T01:24:44.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7669999999999999e-06 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839350+0000", 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2109e-05 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839372+0000", 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.162e-05 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839396+0000", 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3898999999999999e-05 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986378+0000", 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14698248899999999 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:44.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986395+0000", 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6545000000000002e-05 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.196668+0000", 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.210273194 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.196889+0000", 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022093199999999999 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.626756+0000", 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.42986713199999999 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.635172+0000", 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084160469999999994 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.835968+0000", 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20079548799999999 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.836114+0000", 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014641 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:543 3.ds0 3:b6ed1d05:::smithi18128752-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2017462239999999, 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.79564045900000002, 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.070 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 543 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839437+0000", 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6130000000000001e-06 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839444+0000", 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7700000000000004e-06 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839465+0000", 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0970000000000001e-05 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839486+0000", 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1199000000000002e-05 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986531+0000", 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14704526000000001 2024-04-26T01:24:44.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986557+0000", 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5701999999999999e-05 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197379+0000", 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21082251799999999 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197562+0000", 2024-04-26T01:24:44.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000182652 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.634726+0000", 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43716451899999997 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.634892+0000", 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016576000000000001 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.635076+0000", 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018349600000000001 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.as1 67/63 ECSubWrite(tid=27, reqid=osd.3.0:27, at_version=67'23, trim_to=0'0, roll_forward_to=66'21))", 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.898117+0000", 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.1430639490000001, 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19970385800000001, 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.898117+0000", 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.898117+0000", 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.898118+0000", 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6000000000000003e-07 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.898205+0000", 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6944e-05 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.898239+0000", 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3456e-05 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.898272+0000", 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3036e-05 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096029+0000", 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19775711000000001 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096043+0000", 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4756000000000001e-05 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.097749+0000", 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001705561 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.097821+0000", 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2335000000000005e-05 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDPGRecoveryDeleteReply(3.ds0 e67,63 3:b2e0ecb3:::smithi18128752-14:8,61'32)", 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.096928+0000", 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9442535940000001, 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102030302, 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.080 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096928+0000", 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096928+0000", 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096928+0000", 2024-04-26T01:24:44.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3399999999999999e-07 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096935+0000", 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.849e-06 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096951+0000", 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5362999999999998e-05 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.096969+0000", 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8113000000000001e-05 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.198237+0000", 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101267959 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.198958+0000", 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00072138400000000004 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.083 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s1 67/63 ECSubWrite(tid=260, reqid=client.4589.0:536, at_version=67'5, trim_to=0'0, roll_forward_to=66'4))", 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.097060+0000", 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9441212539999997, 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10254071400000001, 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.097060+0000", 2024-04-26T01:24:44.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.097060+0000", 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.097060+0000", 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2399999999999999e-07 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.097188+0000", 2024-04-26T01:24:44.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000127164 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.097209+0000", 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1182000000000001e-05 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197319+0000", 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010978 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197492+0000", 2024-04-26T01:24:44.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017369900000000001 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197520+0000", 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8005999999999999e-05 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199477+0000", 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019567920000000002 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199601+0000", 2024-04-26T01:24:44.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000123767 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 67/63 ECSubWriteReply(tid=8, last_complete=67'37, committed=1, applied=1))", 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.303616+0000", 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.7375651249999997, 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10205813800000001, 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.303616+0000", 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.303616+0000", 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.303618+0000", 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.75e-06 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.303629+0000", 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0482000000000001e-05 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.303653+0000", 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4406999999999999e-05 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.403744+0000", 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009094 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.403939+0000", 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000195016 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.405674+0000", 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001735543 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 67/63 ECSubWriteReply(tid=9, last_complete=67'37, committed=1, applied=1))", 2024-04-26T01:24:44.091 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.407579+0000", 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.633602915, 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10115399799999999, 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.407579+0000", 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.407579+0000", 2024-04-26T01:24:44.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.407580+0000", 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.813e-06 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.407592+0000", 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2063e-05 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.407620+0000", 2024-04-26T01:24:44.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7506000000000001e-05 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.507714+0000", 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009451 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.508605+0000", 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089040600000000005 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.508732+0000", 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012769999999999999 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 67/63 ECSubWriteReply(tid=10, last_complete=67'39, committed=1, applied=1))", 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.705197+0000", 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.335983959, 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13195691800000001, 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.095 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.705197+0000", 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.705197+0000", 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.705199+0000", 2024-04-26T01:24:44.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6959999999999999e-06 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.705208+0000", 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1800000000000002e-06 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.705236+0000", 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8053000000000001e-05 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.705253+0000", 2024-04-26T01:24:44.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6402e-05 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.835822+0000", 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.13056953499999999 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.837154+0000", 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001332052 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 68/63 ECSubWriteReply(tid=11, last_complete=67'39, committed=1, applied=1))", 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.838716+0000", 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.2024654520000002, 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100693351, 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838716+0000", 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838716+0000", 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838717+0000", 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.23e-07 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838721+0000", 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9110000000000003e-06 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838732+0000", 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1061e-05 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838741+0000", 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7550000000000006e-06 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.939372+0000", 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100631025 2024-04-26T01:24:44.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.939409+0000", 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6875999999999997e-05 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 67/63 ECSubWriteReply(tid=11, last_complete=67'39, committed=1, applied=1))", 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.838814+0000", 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.2023669300000002, 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10104310499999999, 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.102 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838814+0000", 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838814+0000", 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838816+0000", 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9439999999999999e-06 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838827+0000", 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0916e-05 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838855+0000", 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7216000000000002e-05 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.838870+0000", 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.526e-05 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.939830+0000", 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10096017 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.939858+0000", 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7599000000000001e-05 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.105 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 67/63 ECSubWriteReply(tid=11, last_complete=67'39, committed=1, applied=1))", 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.841719+0000", 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1994622140000004, 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19834401600000001, 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.841719+0000", 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.841719+0000", 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.841720+0000", 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9000000000000002e-07 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.841724+0000", 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.284e-06 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.841748+0000", 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4105e-05 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.841765+0000", 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.658e-05 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.039992+0000", 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.198227449 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.040063+0000", 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1208e-05 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s5 72/63 ECSubWrite(tid=54, reqid=osd.0.0:54, at_version=72'37, trim_to=0'0, roll_forward_to=71'35))", 2024-04-26T01:24:44.109 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:43.869855+0000", 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.17132642300000001, 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10121287800000001, 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.869855+0000", 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.869855+0000", 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.869855+0000", 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2100000000000001e-07 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.869884+0000", 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9050000000000001e-05 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.869901+0000", 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6361000000000001e-05 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.869914+0000", 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3312000000000001e-05 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.970062+0000", 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100147887 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.970075+0000", 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3638000000000001e-05 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.971024+0000", 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00094879600000000001 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.971068+0000", 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3612999999999997e-05 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 72/63 ECSubWriteReply(tid=26, last_complete=71'56, committed=1, applied=1))", 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:43.919574+0000", 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.12160789499999999, 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10090560599999999, 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.919574+0000", 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.919574+0000", 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.919581+0000", 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7370000000000006e-06 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.919591+0000", 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3230000000000004e-06 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.919615+0000", 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4439000000000001e-05 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.919629+0000", 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4053000000000001e-05 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020458+0000", 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100828973 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020479+0000", 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1081e-05 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.117 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:44.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:24:44.119 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 89 (ObjNum 69 snap 6 seq_num 69) dirty exists 2024-04-26T01:24:44.124 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:44.125 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:44.160 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:44.165+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:44.166 INFO:tasks.rados.rados.0.smithi181.stdout:242: finishing copy_from to smithi18128752-48 2024-04-26T01:24:44.166 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 14 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-26T01:24:44.217 INFO:tasks.rados.rados.0.smithi181.stdout:241: finishing write tid 6 to smithi18128752-47 2024-04-26T01:24:44.218 INFO:tasks.rados.rados.0.smithi181.stdout:241: oid 47 updating version 0 to 72 2024-04-26T01:24:44.218 INFO:tasks.rados.rados.0.smithi181.stdout:241: oid 47 updating version 72 to 73 2024-04-26T01:24:44.218 INFO:tasks.rados.rados.0.smithi181.stdout:241: oid 47 updating version 73 to 75 2024-04-26T01:24:44.218 INFO:tasks.rados.rados.0.smithi181.stdout:241: oid 47 version 75 is already newer than 74 2024-04-26T01:24:44.218 INFO:tasks.rados.rados.0.smithi181.stdout:241: oid 47 updating version 75 to 76 2024-04-26T01:24:44.218 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 47 v 76 (ObjNum 97 snap 17 seq_num 97) dirty exists 2024-04-26T01:24:44.218 INFO:tasks.rados.rados.0.smithi181.stdout:241: left oid 47 (ObjNum 97 snap 17 seq_num 97) 2024-04-26T01:24:44.226 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:44.229+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 72/34 ECSubWriteReply(tid=108, last_complete=72'79, committed=1, applied=1))", 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.220245+0000", 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.074168810000000002, 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.074209575999999999, 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:44.289 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.220245+0000", 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.220245+0000", 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.220245+0000", 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1100000000000005e-07 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.220249+0000", 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8809999999999998e-06 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.220261+0000", 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2044e-05 2024-04-26T01:24:44.291 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.292 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.292 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.292 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.292 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-26T01:24:44.292 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2024-04-26T01:24:44.292 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:44.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:24:44.312 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 1 to smithi18128752-28 2024-04-26T01:24:44.313 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 2 to smithi18128752-28 2024-04-26T01:24:44.355 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:44.361+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:44.414 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 3 to smithi18128752-28 2024-04-26T01:24:44.414 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 4 to smithi18128752-28 2024-04-26T01:24:44.415 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 5 to smithi18128752-28 2024-04-26T01:24:44.424 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:44.425 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:44.427 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:44.427+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:44.431 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:44.431 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:44.431 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:44.431 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:44.431 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:44.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:24:44.469 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:44.470 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:44.528 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:44.531+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:44.569 INFO:tasks.rados.rados.0.smithi181.stdout:247: finishing write tid 1 to smithi18128752-43 2024-04-26T01:24:44.569 INFO:tasks.rados.rados.0.smithi181.stdout:247: finishing write tid 2 to smithi18128752-43 2024-04-26T01:24:44.613 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:150 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.438809591000002, 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3959368480000001, 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.614 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 150 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:24:44.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179064+0000", 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7269999999999999e-06 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179608+0000", 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00054408799999999997 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179641+0000", 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2873000000000002e-05 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179658+0000", 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6863000000000001e-05 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268330+0000", 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0886726819999999 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268562+0000", 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023123100000000001 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.270316+0000", 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017541189999999999 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474438+0000", 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20412262 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574871+0000", 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100432788 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574998+0000", 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126857 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:151 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:44.619 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.432434729000001, 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.494027684, 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 151 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:44.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185438+0000", 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2170000000000001e-06 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185895+0000", 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000456291 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185927+0000", 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1989000000000002e-05 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185942+0000", 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5075999999999999e-05 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272106+0000", 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0861648319999999 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272243+0000", 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013630399999999999 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.273909+0000", 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016665930000000001 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474512+0000", 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20060244399999999 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679353+0000", 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20484124000000001 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679464+0000", 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110698 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.624 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:152 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.431902823000001, 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4937803999999999, 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 152 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.625 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185969+0000", 2024-04-26T01:24:44.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.167e-06 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185975+0000", 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5029999999999997e-06 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185986+0000", 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1283e-05 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185994+0000", 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6060000000000003e-06 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372230+0000", 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.186235766 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372376+0000", 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014574 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.373451+0000", 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010752890000000001 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474514+0000", 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101062945 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679632+0000", 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20511794799999999 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679748+0000", 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116153 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:153 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.431864447000002, 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.900776316, 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.630 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 153 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:24:44.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186007+0000", 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6700000000000002e-07 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186011+0000", 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5559999999999999e-06 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186025+0000", 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3873000000000001e-05 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186035+0000", 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0526000000000001e-05 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475079+0000", 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2890438200000001 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:44.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475255+0000", 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000176279 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680464+0000", 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20520917499999999 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680595+0000", 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013023099999999999 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086783+0000", 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40618798900000003 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.159207582999997, 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.407182076, 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.635 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458663+0000", 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458664+0000", 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458680+0000", 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5953000000000001e-05 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458693+0000", 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2972999999999999e-05 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.458701+0000", 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4829999999999999e-06 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.865517+0000", 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40681550500000002 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.865845+0000", 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00032856200000000002 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:164 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.319957985000002, 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78562828299999998, 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.639 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297916+0000", 2024-04-26T01:24:44.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297932+0000", 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2542e-05 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297993+0000", 2024-04-26T01:24:44.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7813e-05 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475329+0000", 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17733589199999999 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475446+0000", 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117074 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083541+0000", 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60809543499999996 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:165 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.319842571000002, 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78769006399999997, 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:24:44.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298030+0000", 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.59e-06 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298039+0000", 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1470000000000007e-06 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298064+0000", 2024-04-26T01:24:44.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5542e-05 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298080+0000", 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5279999999999999e-05 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476021+0000", 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.177941503 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476177+0000", 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155873 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085718+0000", 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60954112900000001 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:166 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.319766530000003, 2024-04-26T01:24:44.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78831037500000001, 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 166 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298106+0000", 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-06 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298113+0000", 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7570000000000004e-06 2024-04-26T01:24:44.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298128+0000", 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5e-05 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298139+0000", 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0986e-05 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476693+0000", 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17855336099999999 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476835+0000", 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014191599999999999 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086415+0000", 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60957998499999999 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/34 ECSubReadReply(tid=23, attrs_read=0))", 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.017248504000001, 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.48295446400000003, 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600624+0000", 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6199999999999999e-06 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601039+0000", 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041512500000000002 2024-04-26T01:24:44.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601068+0000", 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.906e-05 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701180+0000", 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10011215699999999 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.082862+0000", 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.381681457 2024-04-26T01:24:44.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083577+0000", 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071504499999999998 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:189 3.cs0 3:395c8678:::smithi18128752-25:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.540830917999997, 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61527076999999997, 2024-04-26T01:24:44.655 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077044+0000", 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0570000000000003e-06 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077061+0000", 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6616000000000002e-05 2024-04-26T01:24:44.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077097+0000", 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6829000000000003e-05 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077120+0000", 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2779999999999999e-05 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086851+0000", 2024-04-26T01:24:44.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097310599999999997 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.087007+0000", 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015570199999999999 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.495435+0000", 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40842789299999999 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497946+0000", 2024-04-26T01:24:44.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025112519999999998 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692090+0000", 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19414421300000001 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692311+0000", 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220368 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:190 3.cs0 3:395c8678:::smithi18128752-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.540714780999998, 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.42078353899999998, 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190 2024-04-26T01:24:44.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077158+0000", 2024-04-26T01:24:44.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7540000000000001e-06 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077167+0000", 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2459999999999992e-06 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077188+0000", 2024-04-26T01:24:44.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1403000000000002e-05 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077357+0000", 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000168985 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187444+0000", 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110086218 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:44.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187592+0000", 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014808599999999999 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497585+0000", 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.309993713 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497788+0000", 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020263099999999999 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497940+0000", 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015150299999999999 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:199 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.540471080000003, 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41377173299999997, 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.956e-06 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077413+0000", 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1056999999999999e-05 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077435+0000", 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2602000000000001e-05 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7252999999999999e-05 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187721+0000", 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110268303 2024-04-26T01:24:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187913+0000", 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000191845 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.491171+0000", 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30325871700000001 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:200 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.540394096, 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41453494499999999, 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 200 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.364e-06 2024-04-26T01:24:44.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077486+0000", 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8159999999999994e-06 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077505+0000", 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9360000000000001e-05 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077521+0000", 2024-04-26T01:24:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5685999999999999e-05 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188598+0000", 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111076802 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188834+0000", 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023592099999999999 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.492012+0000", 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30317799600000001 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:201 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.540330804, 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41550727300000001, 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.675 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 201 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:24:44.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077541+0000", 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.314e-06 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077548+0000", 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8390000000000001e-06 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077566+0000", 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7893e-05 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077581+0000", 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5290000000000001e-05 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.189776+0000", 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112194944 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.190248+0000", 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047181099999999999 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.493047+0000", 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.302799182 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:321 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.149726+0000", 2024-04-26T01:24:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.468145100000001, 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.405632877, 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 321 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149726+0000", 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149726+0000", 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149729+0000", 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9000000000000002e-06 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149741+0000", 2024-04-26T01:24:44.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2862e-05 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149773+0000", 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1983999999999997e-05 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149791+0000", 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7422e-05 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.234698+0000", 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.084906827000000004 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.234724+0000", 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6466e-05 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.350693+0000", 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.115968881 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.350993+0000", 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000299654 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.555359+0000", 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.204365881 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:322 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:24:44.684 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.468049748999999, 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.707643833, 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 322 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149822+0000", 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.004e-06 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149829+0000", 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.613e-06 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149840+0000", 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1819e-05 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249944+0000", 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100103704 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250099+0000", 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015437400000000001 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250119+0000", 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0392999999999999e-05 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352060+0000", 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101941083 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352219+0000", 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015886700000000001 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857465+0000", 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50524597599999999 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:323 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:44.689 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.367887938999999, 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.70925194000000003, 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 323 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249984+0000", 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4559999999999999e-06 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249994+0000", 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0186e-05 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250012+0000", 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7836999999999999e-05 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250023+0000", 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1143e-05 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254128+0000", 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041046069999999997 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254145+0000", 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7405999999999999e-05 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452144+0000", 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.197998339 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452432+0000", 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028853800000000002 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.959235+0000", 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50680242799999997 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.694 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s6 47/30 ECSubWrite(tid=123, reqid=client.4589.0:325, at_version=47'28, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.161103422, 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50042386699999997, 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456768+0000", 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7599999999999997e-07 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456800+0000", 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2240000000000003e-05 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456819+0000", 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8799000000000001e-05 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456835+0000", 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5860000000000001e-05 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.655702+0000", 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19886722300000001 2024-04-26T01:24:44.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.655714+0000", 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.218e-05 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957130+0000", 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301416354 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957191+0000", 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0735e-05 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s6 47/30 ECSubWrite(tid=125, reqid=client.4589.0:326, at_version=47'29, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.156865658000001, 2024-04-26T01:24:44.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.496226999, 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461007+0000", 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7689999999999999e-06 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461040+0000", 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3563999999999999e-05 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461069+0000", 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8989e-05 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.561206+0000", 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10013631100000001 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857800+0000", 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29659451599999997 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857828+0000", 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7583e-05 2024-04-26T01:24:44.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957201+0000", 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099372708000000004 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957232+0000", 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1559000000000001e-05 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:397 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [copy-from ver 14] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.377072169000002, 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.58707355400000005, 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 397 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.704 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240802+0000", 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5200000000000002e-06 2024-04-26T01:24:44.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240819+0000", 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6444999999999999e-05 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240855+0000", 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6702999999999997e-05 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:44.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240873+0000", 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7900000000000001e-05 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.241008+0000", 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013451000000000001 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.241021+0000", 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3504e-05 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.245054+0000", 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004032926 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.245220+0000", 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016579 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.553412+0000", 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30819260700000001 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.556948+0000", 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035356580000000001 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.827661+0000", 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.270712428 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.827872+0000", 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021156300000000001 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.710 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.710 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.710 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 6 to smithi18128752-28 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 28 updating version 0 to 22 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 28 version 22 is already newer than 19 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 28 updating version 22 to 23 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 28 version 23 is already newer than 21 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 28 version 23 is already newer than 20 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 28 v 23 (ObjNum 101 snap 17 seq_num 101) dirty exists 2024-04-26T01:24:44.711 INFO:tasks.rados.rados.0.smithi181.stdout:248: left oid 28 (ObjNum 101 snap 17 seq_num 101) 2024-04-26T01:24:44.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:24:44.713 INFO:tasks.rados.rados.0.smithi181.stdout:249: finishing copy_from racing read to smithi18128752-30 2024-04-26T01:24:44.713 INFO:tasks.rados.rados.0.smithi181.stdout:247: finishing write tid 3 to smithi18128752-43 2024-04-26T01:24:44.713 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:44.715+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:44.772 INFO:tasks.rados.rados.0.smithi181.stdout:247: finishing write tid 4 to smithi18128752-43 2024-04-26T01:24:44.788 INFO:tasks.rados.rados.0.smithi181.stdout:241: done (6 left) 2024-04-26T01:24:44.788 INFO:tasks.rados.rados.0.smithi181.stdout:242: done (5 left) 2024-04-26T01:24:44.788 INFO:tasks.rados.rados.0.smithi181.stdout:245: done (4 left) 2024-04-26T01:24:44.788 INFO:tasks.rados.rados.0.smithi181.stdout:248: done (3 left) 2024-04-26T01:24:44.788 INFO:tasks.rados.rados.0.smithi181.stdout:250: done (2 left) 2024-04-26T01:24:44.788 INFO:tasks.rados.rados.0.smithi181.stdout:251: snap_create 2024-04-26T01:24:44.812 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:44.815+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:44.841 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:44.842 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:44.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.842 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 72/44 ECSubWrite(tid=197, reqid=osd.14.0:196, at_version=0'0, trim_to=0'0, roll_forward_to=72'93))", 2024-04-26T01:24:44.842 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:24:44.842 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.066717611999999996, 2024-04-26T01:24:44.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.066756825000000006, 2024-04-26T01:24:44.842 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780912+0000", 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.373e-06 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780936+0000", 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3869000000000001e-05 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780967+0000", 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0565999999999998e-05 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:44.844 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.845 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:44.845 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-26T01:24:44.845 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2024-04-26T01:24:44.845 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:44.851 INFO:tasks.rados.rados.0.smithi181.stdout:249: finishing copy_from to smithi18128752-30 2024-04-26T01:24:44.851 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 59 (ObjNum 92 snap 17 seq_num 92) dirty exists 2024-04-26T01:24:44.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:24:44.876 INFO:tasks.rados.rados.0.smithi181.stdout:247: finishing write tid 5 to smithi18128752-43 2024-04-26T01:24:44.876 INFO:tasks.rados.rados.0.smithi181.stdout:247: oid 43 updating version 0 to 92 2024-04-26T01:24:44.876 INFO:tasks.rados.rados.0.smithi181.stdout:247: oid 43 version 92 is already newer than 91 2024-04-26T01:24:44.876 INFO:tasks.rados.rados.0.smithi181.stdout:247: oid 43 version 92 is already newer than 90 2024-04-26T01:24:44.876 INFO:tasks.rados.rados.0.smithi181.stdout:247: oid 43 version 92 is already newer than 89 2024-04-26T01:24:44.877 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 92 (ObjNum 100 snap 17 seq_num 100) dirty exists 2024-04-26T01:24:44.877 INFO:tasks.rados.rados.0.smithi181.stdout:247: left oid 43 (ObjNum 100 snap 17 seq_num 100) 2024-04-26T01:24:44.903 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:24:44.912 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:44.917+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:44.975 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:44.975 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:44.975 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:44.975 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:44.975 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:44.991 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:24:45.013 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:45.015+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:45.114 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:45.115+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:45.156 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:45.156 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:45.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:45.156 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:45.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 37/34 ECSubWrite(tid=19, reqid=client.4589.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=37'11))", 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 36.878026783000003, 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103193194, 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281194+0000", 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281196+0000", 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.046e-06 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281226+0000", 2024-04-26T01:24:45.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9762999999999999e-05 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281269+0000", 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3059000000000002e-05 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281295+0000", 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6432999999999999e-05 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281433+0000", 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013830900000000001 2024-04-26T01:24:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.281449+0000", 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5846999999999999e-05 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384326+0000", 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102877183 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.384387+0000", 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0554000000000003e-05 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.160 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 32.663225976, 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10316732200000001, 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495995+0000", 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.495996+0000", 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.457e-06 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496024+0000", 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8269999999999999e-05 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496047+0000", 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3045e-05 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.496077+0000", 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0046999999999999e-05 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.596334+0000", 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10025700899999999 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.596353+0000", 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8423e-05 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.599074+0000", 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002720943 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.599162+0000", 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8127999999999994e-05 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s3 43/34 ECSubWrite(tid=30, reqid=osd.0.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=43'19))", 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:14.315851+0000", 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.843369175999999, 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103015009, 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.165 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315851+0000", 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315851+0000", 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315853+0000", 2024-04-26T01:24:45.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9970000000000001e-06 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315886+0000", 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2526e-05 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.315928+0000", 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2502000000000001e-05 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.416043+0000", 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100114561 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.416195+0000", 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151793 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.416223+0000", 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7838999999999999e-05 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.418768+0000", 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025458199999999999 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:14.418866+0000", 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7971000000000001e-05 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.169 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 47/30 ECSubWrite(tid=74, reqid=client.4589.0:332, at_version=47'15, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.584052363000001, 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20225293799999999, 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.170 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575170+0000", 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.08e-06 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575220+0000", 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0246e-05 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575252+0000", 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1482000000000001e-05 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.675393+0000", 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014118599999999 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775690+0000", 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029639899999999 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775706+0000", 2024-04-26T01:24:45.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6133e-05 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777345+0000", 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016393899999999999 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777421+0000", 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6021999999999998e-05 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.174 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=58, reqid=client.4589.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.559559405000002, 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301426421, 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599662+0000", 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.299e-06 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599675+0000", 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2605e-05 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599692+0000", 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7221999999999999e-05 2024-04-26T01:24:45.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699796+0000", 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100104162 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699955+0000", 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000158319 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699983+0000", 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8453e-05 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901051+0000", 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201067725 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901088+0000", 2024-04-26T01:24:45.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6636e-05 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=60, reqid=osd.5.0:60, at_version=48'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.459275988999998, 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20186222600000001, 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.180 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9200000000000002e-07 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699973+0000", 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8328999999999999e-05 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699980+0000", 2024-04-26T01:24:45.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0770000000000002e-06 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699990+0000", 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.011e-05 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900620+0000", 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20062928899999999 2024-04-26T01:24:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900633+0000", 2024-04-26T01:24:45.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2994e-05 2024-04-26T01:24:45.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901770+0000", 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001137482 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901807+0000", 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6653000000000003e-05 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 51/44 ECSubWrite(tid=67, reqid=client.4589.0:382, at_version=51'41, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.943334968999999, 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103342327, 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.215886+0000", 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3700000000000001e-07 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.216245+0000", 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00035873300000000002 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.216275+0000", 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0528999999999999e-05 2024-04-26T01:24:45.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.216293+0000", 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7827000000000001e-05 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.316559+0000", 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100266483 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.316576+0000", 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6328999999999999e-05 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.319160+0000", 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025844570000000001 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.319228+0000", 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7632000000000006e-05 2024-04-26T01:24:45.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 61/44 ECSubWrite(tid=124, reqid=client.4589.0:468, at_version=61'53, trim_to=0'0, roll_forward_to=59'47))", 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:32.945195+0000", 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 12.214025453, 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10418532899999999, 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.192 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:32.945195+0000", 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:32.945195+0000", 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:32.945197+0000", 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3039999999999999e-06 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:32.946033+0000", 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083562399999999998 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:32.946076+0000", 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3004999999999999e-05 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.046182+0000", 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010601399999999 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.046309+0000", 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012689 2024-04-26T01:24:45.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.046325+0000", 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5889999999999999e-05 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.049329+0000", 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030045639999999999 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:33.049380+0000", 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1038000000000002e-05 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 66/30 ECSubWrite(tid=142, reqid=client.4589.0:523, at_version=66'28, trim_to=0'0, roll_forward_to=61'24))", 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.087206+0000", 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.0720147310000003, 2024-04-26T01:24:45.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10301202, 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087206+0000", 2024-04-26T01:24:45.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087206+0000", 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087207+0000", 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9699999999999994e-07 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087306+0000", 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9258000000000004e-05 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087325+0000", 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8995999999999999e-05 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087336+0000", 2024-04-26T01:24:45.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1353e-05 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.188374+0000", 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101037764 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.188394+0000", 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0040000000000001e-05 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.190123+0000", 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017290140000000001 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.190218+0000", 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4598000000000005e-05 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.202 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 66/30 ECSubWrite(tid=143, reqid=client.4589.0:524, at_version=66'29, trim_to=0'0, roll_forward_to=61'24))", 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.087361+0000", 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.0718594709999998, 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.105147927, 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087361+0000", 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087361+0000", 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087362+0000", 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2399999999999996e-07 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087409+0000", 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7104999999999997e-05 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087425+0000", 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5800000000000001e-05 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.087437+0000", 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2262999999999999e-05 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.189226+0000", 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10178933900000001 2024-04-26T01:24:45.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.189258+0000", 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1362999999999998e-05 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.192448+0000", 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031906080000000002 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.192509+0000", 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0525000000000001e-05 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=146, reqid=osd.14.0:146, at_version=67'60, trim_to=0'0, roll_forward_to=61'58))", 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.3680119260000003, 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200985516, 2024-04-26T01:24:45.207 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:24:45.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3e-07 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791259+0000", 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9898999999999998e-05 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791388+0000", 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012909999999999999 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.891494+0000", 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010556600000001 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.990490+0000", 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098996585999999998 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.990520+0000", 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0076000000000001e-05 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.992124+0000", 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016037709999999999 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.992194+0000", 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9888000000000006e-05 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.212 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 67/44 ECSubWrite(tid=113, reqid=osd.5.0:113, at_version=67'78, trim_to=0'0, roll_forward_to=67'76))", 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0193177630000001, 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201983458, 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.213 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139905+0000", 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.86e-06 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139946+0000", 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0948999999999997e-05 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139975+0000", 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9130000000000001e-05 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139990+0000", 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5712000000000001e-05 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.238557+0000", 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098566661999999999 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.238587+0000", 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9757e-05 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341833+0000", 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103246136 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341886+0000", 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3251999999999998e-05 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 67/44 ECSubWrite(tid=114, reqid=osd.5.0:114, at_version=67'80, trim_to=0'0, roll_forward_to=67'76))", 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0170852650000004, 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199785569, 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:24:45.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142137+0000", 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.843e-06 2024-04-26T01:24:45.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142182+0000", 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4796000000000003e-05 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142211+0000", 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9394999999999999e-05 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142227+0000", 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5630000000000001e-05 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239528+0000", 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097300638999999994 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239543+0000", 2024-04-26T01:24:45.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5616000000000001e-05 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341893+0000", 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102349381 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341921+0000", 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8269e-05 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.223 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=150, reqid=osd.14.0:147, at_version=0'0, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.9633841519999997, 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.146116522, 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195838+0000", 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.03e-06 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195859+0000", 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0988999999999998e-05 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195891+0000", 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1962999999999999e-05 2024-04-26T01:24:45.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195911+0000", 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0046000000000001e-05 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239504+0000", 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043592858999999998 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239529+0000", 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5086999999999999e-05 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341923+0000", 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102394075 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341953+0000", 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9473000000000001e-05 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=67'64, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:24:45.229 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.9613825689999995, 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14414429500000001, 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197840+0000", 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7269999999999999e-06 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197895+0000", 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5092e-05 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197934+0000", 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8846e-05 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197958+0000", 2024-04-26T01:24:45.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3989e-05 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.340310+0000", 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.142352647 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.340336+0000", 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5992000000000001e-05 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341958+0000", 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016213429999999999 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341982+0000", 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4658999999999999e-05 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=152, reqid=osd.14.0:152, at_version=67'66, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.9597139480000001, 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.142969076, 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199509+0000", 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1270000000000001e-06 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199563+0000", 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4565999999999999e-05 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199599+0000", 2024-04-26T01:24:45.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6168999999999999e-05 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199623+0000", 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3275e-05 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341198+0000", 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14157494500000001 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341226+0000", 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.342404+0000", 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011773689999999999 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.342476+0000", 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1995000000000004e-05 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s3 67/63 ECSubWrite(tid=267, reqid=osd.13.0:265, at_version=0'0, trim_to=0'0, roll_forward_to=67'7))", 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.4566511179999999, 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20128004799999999, 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.240 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702572+0000", 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.52e-06 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702594+0000", 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2166000000000001e-05 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702623+0000", 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8960000000000001e-05 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.802711+0000", 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100088178 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.902926+0000", 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10021514500000001 2024-04-26T01:24:45.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.902945+0000", 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8510000000000001e-05 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.903760+0000", 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081536499999999997 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.903849+0000", 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9203999999999995e-05 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=156, reqid=osd.14.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3585177990000004, 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.302725417, 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.245 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800704+0000", 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6869999999999999e-06 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800726+0000", 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1379e-05 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800752+0000", 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6472999999999999e-05 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800766+0000", 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3441999999999999e-05 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.901012+0000", 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100246591 2024-04-26T01:24:45.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.901028+0000", 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.611e-05 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103277+0000", 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20224862199999999 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103428+0000", 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151113 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=157, reqid=osd.14.0:157, at_version=67'68, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.3559810790000002, 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30024277599999999, 2024-04-26T01:24:45.250 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:24:45.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803247+0000", 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5569999999999998e-06 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803288+0000", 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0778999999999997e-05 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803309+0000", 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1342999999999999e-05 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803326+0000", 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7214000000000002e-05 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.001773+0000", 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19844687899999999 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.001789+0000", 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5449999999999999e-05 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103444+0000", 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101655572 2024-04-26T01:24:45.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103482+0000", 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7982000000000003e-05 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=158, reqid=osd.14.0:158, at_version=67'70, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:24:45.255 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.353521229, 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29805379300000001, 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:45.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805700+0000", 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.105e-06 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805744+0000", 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4044e-05 2024-04-26T01:24:45.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805766+0000", 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1772000000000001e-05 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805778+0000", 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.166e-05 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.102518+0000", 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.296740585 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.102534+0000", 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5571999999999999e-05 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103634+0000", 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011001979999999999 2024-04-26T01:24:45.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103753+0000", 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118857 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:45.260 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:45.262 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:45.265+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:45.262 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:24:45.363 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:45.367+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:45.385 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:45.385 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:45.385 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:45.385 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:45.399 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:24:45.463 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:45.467+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:45.519 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:45.520 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:45.520 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:45.520 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:45.520 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:45.535 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:24:45.583 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:45.587+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:115 3.6s0 3:69f31b93:::smithi18128752-39:head [append 0~638976,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:45.659 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.275141613999999, 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.27460739099999998, 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 115 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:45.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385703+0000", 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.385705+0000", 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5990000000000001e-06 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386759+0000", 2024-04-26T01:24:45.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010545940000000001 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386792+0000", 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2376999999999998e-05 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386816+0000", 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4680000000000001e-05 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.454087+0000", 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.067270658999999997 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.454285+0000", 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019791700000000001 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.455558+0000", 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001272731 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558702+0000", 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103144082 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.659941+0000", 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101238869 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.660311+0000", 2024-04-26T01:24:45.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000369883 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:116 3.6s0 3:69f31b93:::smithi18128752-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.273989868000001, 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.37357116200000001, 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.665 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 116 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386857+0000", 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.703e-06 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386866+0000", 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5670000000000001e-06 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386887+0000", 2024-04-26T01:24:45.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1316000000000001e-05 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386904+0000", 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6767000000000001e-05 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.556964+0000", 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17005975700000001 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.557176+0000", 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021186600000000001 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558110+0000", 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00093395700000000002 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.559561+0000", 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014508430000000001 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.759894+0000", 2024-04-26T01:24:45.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033284600000001 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.760426+0000", 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00053253999999999997 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:117 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.273915215000002, 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77605625899999997, 2024-04-26T01:24:45.670 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 117 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386931+0000", 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3599999999999999e-06 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386939+0000", 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8369999999999997e-06 2024-04-26T01:24:45.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386957+0000", 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8683000000000001e-05 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386974+0000", 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558747+0000", 2024-04-26T01:24:45.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17177335099999999 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558933+0000", 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018625099999999999 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761103+0000", 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202169231 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761231+0000", 2024-04-26T01:24:45.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000128734 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162986+0000", 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.401754422 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:156 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~475136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 39.163209471999998, 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33900037700000002, 2024-04-26T01:24:45.675 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 156 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497638+0000", 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2110000000000001e-06 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498467+0000", 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082923399999999998 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498486+0000", 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9341e-05 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498496+0000", 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8179999999999999e-06 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498610+0000", 2024-04-26T01:24:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000114259 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498770+0000", 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016017599999999999 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726262+0000", 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22749161900000001 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726429+0000", 2024-04-26T01:24:45.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000166825 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.728061+0000", 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016325490000000001 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735773+0000", 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0077114189999999997 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:45.680 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836407+0000", 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10063380399999999 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836636+0000", 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022912200000000001 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:157 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 39.156989006000003, 2024-04-26T01:24:45.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.43381848000000001, 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 157 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503857+0000", 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2920000000000001e-06 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504200+0000", 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034286599999999999 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504218+0000", 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7961e-05 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504227+0000", 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6240000000000008e-06 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504381+0000", 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015489900000000001 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504555+0000", 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017342900000000001 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.729729+0000", 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225174034 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.730038+0000", 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030876200000000002 2024-04-26T01:24:45.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.732244+0000", 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022065190000000001 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735776+0000", 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035319330000000001 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937311+0000", 2024-04-26T01:24:45.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201534612 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937674+0000", 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00036354900000000001 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:158 3.6s0 3:60f0095a:::smithi18128752-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 39.156597933999997, 2024-04-26T01:24:45.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.630196439, 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 158 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504255+0000", 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0989999999999999e-06 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504262+0000", 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5660000000000003e-06 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504266+0000", 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0980000000000004e-06 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504650+0000", 2024-04-26T01:24:45.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00038378400000000002 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504796+0000", 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014592199999999999 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733685+0000", 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22888902699999999 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733975+0000", 2024-04-26T01:24:45.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029027399999999998 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.734990+0000", 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010148500000000001 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836708+0000", 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10171771 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134326+0000", 2024-04-26T01:24:45.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29761846199999997 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134443+0000", 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117142 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:159 3.6s0 3:60f0095a:::smithi18128752-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 39.156572308999998, 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.73307043299999997, 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.693 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 159 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504273+0000", 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3500000000000002e-07 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504274+0000", 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.562e-06 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504281+0000", 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3790000000000003e-06 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504286+0000", 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1989999999999997e-06 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504861+0000", 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00057502400000000004 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504981+0000", 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012016899999999999 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837238+0000", 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33225718900000001 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837352+0000", 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113583 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134907+0000", 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29755462599999999 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.135101+0000", 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019402500000000001 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.237343+0000", 2024-04-26T01:24:45.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102242342 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=11, last_complete=35'7, committed=1, applied=1))", 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 38.826698243000003, 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30069599499999999, 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0800000000000003e-07 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834153+0000", 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1699999999999996e-06 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834168+0000", 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5017999999999999e-05 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.934244+0000", 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100075936 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134251+0000", 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200007516 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134843+0000", 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059144700000000004 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.702 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:171 3.6s0 3:69f31b93:::smithi18128752-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 38.324149372000001, 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36489432300000002, 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 171 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336697+0000", 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3829999999999999e-06 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336701+0000", 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.053e-06 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336710+0000", 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8349999999999993e-06 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336715+0000", 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5219999999999998e-06 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337072+0000", 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00035695799999999999 2024-04-26T01:24:45.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337245+0000", 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017283899999999999 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.338075+0000", 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082985399999999999 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.339323+0000", 2024-04-26T01:24:45.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012477390000000001 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701394+0000", 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36207125499999998 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701590+0000", 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000195885 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s3 41/30 ECSubWrite(tid=97, reqid=osd.4.0:97, at_version=41'22, trim_to=0'0, roll_forward_to=39'18))", 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 33.154344731999998, 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201986786, 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506500+0000", 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506501+0000", 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0100000000000001e-06 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506544+0000", 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2605999999999998e-05 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.506567+0000", 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3269999999999999e-05 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.606669+0000", 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10010197899999999 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.706960+0000", 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10029147400000001 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.706979+0000", 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8403e-05 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.708374+0000", 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013951510000000001 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.708487+0000", 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000112893 2024-04-26T01:24:45.711 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 51/44 ECSubWrite(tid=71, reqid=client.4589.0:385, at_version=0'0, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 21.934435920999999, 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30170434200000001, 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726411+0000", 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.88e-06 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726432+0000", 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.137e-05 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726464+0000", 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1664e-05 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826555+0000", 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100091667 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826667+0000", 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011122500000000001 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826683+0000", 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6149000000000001e-05 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028057+0000", 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20137464699999999 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028113+0000", 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5739999999999998e-05 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:430 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801177+0000", 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.859667542, 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.21020058, 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.716 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 430 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801177+0000", 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801177+0000", 2024-04-26T01:24:45.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801181+0000", 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3840000000000001e-06 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801197+0000", 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6008e-05 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801227+0000", 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0701000000000003e-05 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801242+0000", 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4384e-05 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801378+0000", 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013629700000000001 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801391+0000", 2024-04-26T01:24:45.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2628e-05 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.904897+0000", 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103506743 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.905064+0000", 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016662900000000001 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.011378+0000", 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.106313806 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:431 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.859569542999999, 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40905669700000002, 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:45.721 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 431 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:24:45.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801276+0000", 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1399999999999995e-07 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801283+0000", 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4840000000000001e-06 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801299+0000", 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6707000000000001e-05 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801309+0000", 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0112000000000001e-05 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801486+0000", 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017675199999999999 2024-04-26T01:24:45.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801494+0000", 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6490000000000008e-06 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906250+0000", 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10475599300000001 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906413+0000", 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016362200000000001 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.210332+0000", 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303918464 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:432 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:24:45.726 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.859461615000001, 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51044518999999999, 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 432 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:24:45.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801384+0000", 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801390+0000", 2024-04-26T01:24:45.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2750000000000001e-06 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801402+0000", 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1747e-05 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.901507+0000", 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100104586 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.007942+0000", 2024-04-26T01:24:45.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10643573100000001 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.008127+0000", 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000185007 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.311828+0000", 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303700894 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.651390501000002, 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30115782000000002, 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009456+0000", 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7799999999999999e-06 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010014+0000", 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00055842399999999996 2024-04-26T01:24:45.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010048+0000", 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3407e-05 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010060+0000", 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.254e-05 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310534+0000", 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30047343599999998 2024-04-26T01:24:45.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310612+0000", 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8232999999999994e-05 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.648447439000002, 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.298319318, 2024-04-26T01:24:45.734 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012425+0000", 2024-04-26T01:24:45.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7229999999999998e-05 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012432+0000", 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9160000000000001e-06 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310652+0000", 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29821998799999999 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310717+0000", 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4183999999999994e-05 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.632182097000001, 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.28320577800000002, 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.738 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028664+0000", 2024-04-26T01:24:45.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.266e-06 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028857+0000", 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019285299999999999 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028876+0000", 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9584000000000001e-05 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.128964+0000", 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100087615 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310808+0000", 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18184441400000001 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.311868+0000", 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001060046 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.741 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 71/44 ECSubWrite(tid=122, reqid=osd.5.0:122, at_version=71'86, trim_to=0'0, roll_forward_to=70'84))", 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 2.639285858, 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30196446599999999, 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.29e-07 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021579+0000", 2024-04-26T01:24:45.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0132e-05 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021590+0000", 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0645000000000001e-05 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.121747+0000", 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100156917 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.322084+0000", 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033731900000001 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.322100+0000", 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5780000000000001e-05 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.323445+0000", 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013450269999999999 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.323523+0000", 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8417000000000005e-05 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:603 3.6s0 3:69f31b93:::smithi18128752-39:head [stat,delete] snapc 12=[12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 2.302975682, 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40426880399999998, 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.746 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 603 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:24:45.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357872+0000", 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.207e-06 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357887+0000", 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4443e-05 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357921+0000", 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3954000000000001e-05 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357937+0000", 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5823000000000001e-05 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.358088+0000", 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015174899999999999 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.358258+0000", 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169999 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.359596+0000", 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001337378 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.361439+0000", 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0018429900000000001 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.761928+0000", 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.400489284 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.762138+0000", 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000209977 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s3 71/30 ECSubWrite(tid=257, reqid=osd.4.0:257, at_version=71'80, trim_to=0'0, roll_forward_to=71'76))", 2024-04-26T01:24:45.751 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.901446808, 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30249526599999998, 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:24:45.752 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759399+0000", 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2999999999999999e-07 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759444+0000", 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5158999999999999e-05 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759467+0000", 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2892999999999999e-05 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759485+0000", 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8369999999999999e-05 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.960407+0000", 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200921553 2024-04-26T01:24:45.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.960423+0000", 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.061786+0000", 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101363014 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.061893+0000", 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010717699999999999 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.755 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.756 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.756 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.756 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:45.757 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:24:45.758 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:45.761+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:45.784 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-04-26T01:24:45.790 INFO:tasks.rados.rados.0.smithi181.stdout:247: done (2 left) 2024-04-26T01:24:45.790 INFO:tasks.rados.rados.0.smithi181.stdout:249: done (1 left) 2024-04-26T01:24:45.790 INFO:tasks.rados.rados.0.smithi181.stdout:251: done (0 left) 2024-04-26T01:24:45.790 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 17 current snap is 19 2024-04-26T01:24:45.790 INFO:tasks.rados.rados.0.smithi181.stdout:252: seq_num 102 ranges {2916352=557056,3473408=638976,4112384=524288} 2024-04-26T01:24:45.815 INFO:tasks.rados.rados.0.smithi181.stdout:252: writing smithi18128752-17 from 2916352 to 3473408 tid 1 2024-04-26T01:24:45.820 INFO:tasks.rados.rados.0.smithi181.stdout:252: writing smithi18128752-17 from 3473408 to 4112384 tid 2 2024-04-26T01:24:45.824 INFO:tasks.rados.rados.0.smithi181.stdout:252: writing smithi18128752-17 from 4112384 to 4636672 tid 3 2024-04-26T01:24:45.824 INFO:tasks.rados.rados.0.smithi181.stdout:253: rollback oid 8 current snap is 19 2024-04-26T01:24:45.824 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 8 to 18 2024-04-26T01:24:45.824 INFO:tasks.rados.rados.0.smithi181.stdout:254: rmattr oid 26 current snap is 19 2024-04-26T01:24:45.824 INFO:tasks.rados.rados.0.smithi181.stdout:254: done (2 left) 2024-04-26T01:24:45.824 INFO:tasks.rados.rados.0.smithi181.stdout:255: delete oid 2 current snap is 19 2024-04-26T01:24:45.859 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:45.863+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:45.879 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 7] out_osds: [12, 2] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:24:45.879 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-26T01:24:45.917 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:621 3.7s0 3:e9e07e85:::smithi18128752-17:head [append 0~557056 in=557056b] snapc 14=[14,13] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:45.821568+0000", 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.10133822200000001, 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10137410099999999, 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.918 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 621 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.821568+0000", 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.821568+0000", 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.821571+0000", 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0000000000000001e-06 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.822593+0000", 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010221259999999999 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.822627+0000", 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4137000000000002e-05 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.822643+0000", 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6447999999999999e-05 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.827223+0000", 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0045793530000000004 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.827305+0000", 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1967999999999999e-05 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.828312+0000", 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010072690000000001 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.831269+0000", 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0029568289999999998 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:622 3.7s0 3:e9e07e85:::smithi18128752-17:head [append 0~638976 in=638976b] snapc 14=[14,13] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:45.826291+0000", 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.096614473000000006, 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096707457999999996, 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 622 2024-04-26T01:24:45.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.826291+0000", 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.826291+0000", 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.925 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.826293+0000", 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.756e-06 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.827143+0000", 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00084962099999999995 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.827164+0000", 2024-04-26T01:24:45.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1356999999999999e-05 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.827176+0000", 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2057e-05 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.831949+0000", 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0047727999999999998 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.832082+0000", 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013271299999999999 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.833070+0000", 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00098853599999999993 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:45.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.834514+0000", 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00144349 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:623 3.7s0 3.a17e0797 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:45.830087+0000", 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.092819110999999996, 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.092963277999999996, 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.929 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 623 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.830087+0000", 2024-04-26T01:24:45.930 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.830087+0000", 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.830089+0000", 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.869e-06 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.931 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.830693+0000", 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00060470599999999999 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.830714+0000", 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0959e-05 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.932 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.0s6 74/44 ECSubWrite(tid=200, reqid=client.4589.0:626, at_version=74'97, trim_to=0'0, roll_forward_to=72'95))", 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:45.836921+0000", 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.085985268000000004, 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.086159942000000003, 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.836921+0000", 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.836921+0000", 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.836921+0000", 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2800000000000003e-07 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.836987+0000", 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6554e-05 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:45.837003+0000", 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5996000000000001e-05 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T01:24:45.935 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 4 2024-04-26T01:24:45.936 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:45.937 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:24:45.960 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:45.965+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:46.061 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:46.065+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:46.104 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:46.104 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:46.104 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:46.104 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:46.104 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:46.118 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:24:46.128 INFO:tasks.rados.rados.0.smithi181.stdout:253: finishing rollback tid 0 to smithi18128752-8 2024-04-26T01:24:46.129 INFO:tasks.rados.rados.0.smithi181.stdout:255: done (2 left) 2024-04-26T01:24:46.129 INFO:tasks.rados.rados.0.smithi181.stdout:256: rmattr oid 24 current snap is 19 2024-04-26T01:24:46.129 INFO:tasks.rados.rados.0.smithi181.stdout:256: done (2 left) 2024-04-26T01:24:46.129 INFO:tasks.rados.rados.0.smithi181.stdout:257: snap_create 2024-04-26T01:24:46.132 INFO:tasks.rados.rados.0.smithi181.stdout:252: finishing write tid 1 to smithi18128752-17 2024-04-26T01:24:46.132 INFO:tasks.rados.rados.0.smithi181.stdout:252: finishing write tid 2 to smithi18128752-17 2024-04-26T01:24:46.132 INFO:tasks.rados.rados.0.smithi181.stdout:252: finishing write tid 3 to smithi18128752-17 2024-04-26T01:24:46.133 INFO:tasks.rados.rados.0.smithi181.stdout:252: finishing write tid 4 to smithi18128752-17 2024-04-26T01:24:46.138 INFO:tasks.rados.rados.0.smithi181.stdout:253: finishing rollback tid 1 to smithi18128752-8 2024-04-26T01:24:46.162 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:46.167+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:46.232 INFO:tasks.rados.rados.0.smithi181.stdout:253: finishing rollback tid 2 to smithi18128752-8 2024-04-26T01:24:46.232 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 99 (ObjNum 69 snap 6 seq_num 69) dirty exists 2024-04-26T01:24:46.245 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:123 3.ds0 3:b6ed1d05:::smithi18128752-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.754368685000003, 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.401407085, 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.246 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 123 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.247 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493545+0000", 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5299999999999999e-07 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.248 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493546+0000", 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7039999999999999e-06 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493552+0000", 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5119999999999999e-06 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493556+0000", 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2679999999999996e-06 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.249 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697434+0000", 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20387831000000001 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697559+0000", 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012467700000000001 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087644+0000", 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39008522000000001 2024-04-26T01:24:46.250 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087733+0000", 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9191999999999996e-05 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894951+0000", 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80721774899999998 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.251 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:138 3.1s0 3:8830b48d:::smithi18128752-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.468490879999997, 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71477871400000004, 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 138 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.252 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779423+0000", 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.47e-07 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.253 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779429+0000", 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6400000000000002e-06 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779443+0000", 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4392999999999999e-05 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779456+0000", 2024-04-26T01:24:46.254 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2643000000000001e-05 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784323+0000", 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048675369999999999 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784379+0000", 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6314000000000003e-05 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187338+0000", 2024-04-26T01:24:46.255 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40295840700000002 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187417+0000", 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8898000000000003e-05 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.494201+0000", 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30678393500000001 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.256 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:140 3.ds0 3:b9916085:::smithi18128752-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.145035983, 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.69088028800000001, 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 140 2024-04-26T01:24:46.257 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.258 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102878+0000", 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2399999999999997e-07 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102881+0000", 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1489999999999998e-06 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102885+0000", 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1300000000000003e-06 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.259 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102888+0000", 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.242e-06 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290947+0000", 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18805879 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291087+0000", 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013990199999999999 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.260 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291629+0000", 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054169199999999998 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.492101+0000", 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20047221400000001 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.793519+0000", 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30141857300000002 2024-04-26T01:24:46.261 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.793757+0000", 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000237772 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:141 3.ds0 3:b9916085:::smithi18128752-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.145019812000001, 2024-04-26T01:24:46.262 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.99325914900000001, 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 141 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.263 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102896+0000", 2024-04-26T01:24:46.264 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3209999999999999e-06 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102900+0000", 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9369999999999997e-06 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102903+0000", 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3809999999999999e-06 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291929+0000", 2024-04-26T01:24:46.265 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.189026048 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.292027+0000", 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8172999999999997e-05 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794279+0000", 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50225164799999999 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.266 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794416+0000", 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013710400000000001 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.096152+0000", 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30173626199999998 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.267 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.ds0 35/30 ECSubReadReply(tid=43, attrs_read=0))", 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.288989+0000", 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 39.958923368000001, 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60597367199999996, 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.288989+0000", 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.268 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.288989+0000", 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.288991+0000", 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.133e-06 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289001+0000", 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0016e-05 2024-04-26T01:24:46.269 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289010+0000", 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1700000000000003e-06 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.289035+0000", 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4893e-05 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.270 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894810+0000", 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60577542399999995 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894963+0000", 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000153036 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.271 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:325 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.886587242000001, 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70936785199999997, 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 325 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.272 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361328+0000", 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3580000000000001e-06 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.273 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361338+0000", 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.628e-06 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361360+0000", 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2506000000000001e-05 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361374+0000", 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4072e-05 2024-04-26T01:24:46.274 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454039+0000", 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.092664721000000005 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454298+0000", 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025867499999999999 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.275 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.455951+0000", 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016533489999999999 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462924+0000", 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069735650000000001 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070553+0000", 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60762835599999998 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.276 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070693+0000", 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000140622 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:326 3.7s0 3:e977e128:::smithi18128752-21:head [rollback 4] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.886518676000001, 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70968584099999998, 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.277 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 326 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.278 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361395+0000", 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.06e-07 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361398+0000", 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1379999999999999e-06 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.279 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361408+0000", 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.046e-05 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361414+0000", 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8459999999999996e-06 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458365+0000", 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096950878000000004 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.280 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458663+0000", 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000297788 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.460203+0000", 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015404030000000001 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.463428+0000", 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032243850000000002 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.281 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070927+0000", 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.607499821 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071080+0000", 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000152516 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:327 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.886490850000001, 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71007962499999999, 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 327 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.283 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:24:46.284 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361425+0000", 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2740000000000002e-06 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361431+0000", 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4280000000000001e-06 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.285 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361436+0000", 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.994e-06 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461250+0000", 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099813814000000001 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461584+0000", 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000334521 2024-04-26T01:24:46.286 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462509+0000", 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00092412699999999996 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.565364+0000", 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102855424 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.287 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071318+0000", 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50595366500000005 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071502+0000", 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000183912 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:328 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:24:46.288 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.886470446000001, 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81248642000000004, 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 328 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.289 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361443+0000", 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6199999999999998e-07 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.290 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361447+0000", 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.262e-06 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361454+0000", 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8739999999999997e-06 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361459+0000", 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8040000000000001e-06 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.291 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563609+0000", 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20214983 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563788+0000", 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017973199999999999 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.173929+0000", 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.610140456 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.292 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:329 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.88644785, 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81314448500000003, 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 329 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.293 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:24:46.294 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2399999999999999e-07 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361469+0000", 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2820000000000001e-06 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361475+0000", 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3509999999999998e-06 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.295 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361480+0000", 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6959999999999996e-06 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564389+0000", 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202908808 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564593+0000", 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000204255 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.296 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.174609+0000", 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.61001666899999996 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:330 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.886427339000001, 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81407258199999999, 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.297 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 330 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.298 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361486+0000", 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0999999999999999e-07 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361489+0000", 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7240000000000001e-06 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.299 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361495+0000", 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2740000000000004e-06 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361500+0000", 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9400000000000001e-06 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.866935+0000", 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50543468599999997 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.300 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.867233+0000", 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029877499999999998 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.175558+0000", 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.308324773 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:434 3.7s0 3:e9e07e85:::smithi18128752-17:head [copy-from ver 29 in=54b] snapc d=[d,c,8,4] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:46.301 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.46926066, 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1051031140000001, 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 434 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:46.302 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778655+0000", 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8770000000000002e-06 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.303 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778667+0000", 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1888999999999999e-05 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778697+0000", 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0205999999999998e-05 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778714+0000", 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6467999999999999e-05 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.304 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878928+0000", 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100213993 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878940+0000", 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2724999999999999e-05 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882181+0000", 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032411419999999998 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.305 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882351+0000", 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169371 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.678907+0000", 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.79655591199999998 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.882986+0000", 2024-04-26T01:24:46.306 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.204079063 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883081+0000", 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5285000000000007e-05 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883755+0000", 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00067418299999999997 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.307 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:435 3.7s0 3:e9e07e85:::smithi18128752-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 17.469165003000001, 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.003729404, 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 435 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778749+0000", 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.223e-06 2024-04-26T01:24:46.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778755+0000", 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8279999999999999e-06 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778773+0000", 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8289000000000001e-05 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878906+0000", 2024-04-26T01:24:46.310 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10013274699999999 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879035+0000", 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012864199999999999 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879049+0000", 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4137e-05 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.311 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882650+0000", 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036017390000000001 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882716+0000", 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5129000000000006e-05 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782226+0000", 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.89951035899999998 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782341+0000", 2024-04-26T01:24:46.312 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000115492 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782477+0000", 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000135819 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:477 3.bs0 3:d65fea44:::smithi18128752-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 13.355333364, 2024-04-26T01:24:46.313 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.91386205600000003, 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 477 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892580+0000", 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7999999999999996e-07 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892582+0000", 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9630000000000001e-06 2024-04-26T01:24:46.315 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892589+0000", 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7959999999999998e-06 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892593+0000", 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2509999999999996e-06 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.316 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992304+0000", 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099710909 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992356+0000", 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2021000000000002e-05 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402928+0000", 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.41057238600000001 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.317 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.403133+0000", 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020520099999999999 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.806441+0000", 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.403308049 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:537 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~704512] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:24:46.318 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.4207452160000003, 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77667111300000002, 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 537 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:24:46.319 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827172+0000", 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9639999999999997e-06 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.320 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.828324+0000", 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011519550000000001 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.828352+0000", 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8112000000000001e-05 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928473+0000", 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100121081 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.321 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.994151+0000", 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.065677997000000002 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.994232+0000", 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1688999999999997e-05 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.995096+0000", 2024-04-26T01:24:46.322 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00086366299999999997 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997758+0000", 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0026614830000000001 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.603693+0000", 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60593560000000002 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.323 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.603839+0000", 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014556899999999999 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:538 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~688128] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.928521+0000", 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.3193919279999999, 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.67561050700000003, 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.324 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 538 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928521+0000", 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928521+0000", 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.325 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928523+0000", 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6449999999999999e-06 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929086+0000", 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00056332400000000003 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.326 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929102+0000", 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5951999999999999e-05 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929110+0000", 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2730000000000002e-06 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.996156+0000", 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.067045503000000006 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.327 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.996259+0000", 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000103255 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997239+0000", 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00098057200000000008 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999670+0000", 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024309980000000002 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.328 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.604023+0000", 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60435304599999995 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.604131+0000", 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000107939 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:539 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~81920] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:24:46.329 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.3187920200000001, 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77547855799999998, 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 539 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.330 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5400000000000001e-07 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.331 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929175+0000", 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4033000000000003e-05 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929183+0000", 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7130000000000005e-06 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929187+0000", 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2719999999999999e-06 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.332 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997790+0000", 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.068602814999999998 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997917+0000", 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000126783 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998505+0000", 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00058842300000000003 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.333 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999674+0000", 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011685179999999999 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704445+0000", 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.704771335 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704599+0000", 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000154012 2024-04-26T01:24:46.334 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:540 3.1s0 3:865c3313:::smithi18128752-37:head [setxattr _header (57) in=7b] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.3187189759999995, 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77584385600000005, 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.335 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 540 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.336 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9200000000000002e-07 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929197+0000", 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.661e-06 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929201+0000", 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.194e-06 2024-04-26T01:24:46.337 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929204+0000", 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4020000000000002e-06 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998891+0000", 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.069686039000000005 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998997+0000", 2024-04-26T01:24:46.338 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010671 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999475+0000", 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00047769400000000002 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999988+0000", 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051352600000000004 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:46.339 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704829+0000", 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70484043500000004 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.705038+0000", 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000208803 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:541 3.1s0 3:865c3313:::smithi18128752-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:24:46.340 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.3187037070000001, 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0548109999999999, 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 541 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:24:46.341 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9999999999999999e-07 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.342 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929211+0000", 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.517e-06 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929217+0000", 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0410000000000003e-06 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929221+0000", 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.659e-06 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.343 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.028709+0000", 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099488691000000004 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.028788+0000", 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8268000000000003e-05 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.707449+0000", 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.67866140799999997 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.344 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.707686+0000", 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023745600000000001 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.984020+0000", 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.27633365999999998 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(osd.8.22:1 3.7s0 3:e977e128:::smithi18128752-21:head [assert-version v59,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:46.345 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.931199+0000", 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 2.3167134119999999, 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59964591599999995, 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.931199+0000", 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.346 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.931199+0000", 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.931202+0000", 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3070000000000001e-06 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.931214+0000", 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.222e-05 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.347 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.931234+0000", 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0259999999999999e-05 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.031337+0000", 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100102526 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.031504+0000", 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016707899999999999 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.348 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.031697+0000", 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019310599999999999 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.232217+0000", 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20052014600000001 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.232378+0000", 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000160703 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:46.349 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.530845+0000", 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29846756899999999 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:46.350 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:46.351 INFO:tasks.rados.rados.0.smithi181.stdout:252: finishing write tid 5 to smithi18128752-17 2024-04-26T01:24:46.351 INFO:tasks.rados.rados.0.smithi181.stdout:252: oid 17 updating version 0 to 89 2024-04-26T01:24:46.351 INFO:tasks.rados.rados.0.smithi181.stdout:252: oid 17 updating version 89 to 91 2024-04-26T01:24:46.351 INFO:tasks.rados.rados.0.smithi181.stdout:252: oid 17 version 91 is already newer than 88 2024-04-26T01:24:46.352 INFO:tasks.rados.rados.0.smithi181.stdout:252: oid 17 version 91 is already newer than 90 2024-04-26T01:24:46.352 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 91 (ObjNum 102 snap 19 seq_num 102) dirty exists 2024-04-26T01:24:46.352 INFO:tasks.rados.rados.0.smithi181.stdout:252: left oid 17 (ObjNum 102 snap 19 seq_num 102) 2024-04-26T01:24:46.353 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:24:46.353 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:46.357+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:46.453 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:46.455+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:46.504 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:46.525 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:46.525 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:46.526 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:46.526 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:46.540 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:24:46.554 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:46.559+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:46.647 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:46.647 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:46.655 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:46.655+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:46.667 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:46.668 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:46.688 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:46.689 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:46.710 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:46.710 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:46.720 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:46.720 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:46.720 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:46.720 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:46.720 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:46.727 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:46.727 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:46.734 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:24:46.748 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:46.748 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:46.755 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:46.759+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:46.770 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:46.770 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:46.793 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:46.793 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:46.811 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:46.812 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:46.833 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:46.834 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:46.856 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:46.856 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:46.856 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:46.859+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:46.860 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:46.860 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:46.875 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:24:46.879 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T01:24:46.879 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:46.879 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:46.885 INFO:tasks.rados.rados.0.smithi181.stdout:252: done (2 left) 2024-04-26T01:24:46.885 INFO:tasks.rados.rados.0.smithi181.stdout:253: done (1 left) 2024-04-26T01:24:46.885 INFO:tasks.rados.rados.0.smithi181.stdout:257: done (0 left) 2024-04-26T01:24:46.885 INFO:tasks.rados.rados.0.smithi181.stdout:258: copy_from oid 48 from oid 47 current snap is 20 2024-04-26T01:24:46.885 INFO:tasks.rados.rados.0.smithi181.stdout:259: read oid 34 snap -1 2024-04-26T01:24:46.885 INFO:tasks.rados.rados.0.smithi181.stdout:259: expect deleted 2024-04-26T01:24:46.885 INFO:tasks.rados.rados.0.smithi181.stdout:260: snap_create 2024-04-26T01:24:46.908 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-26T01:24:46.909 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:46.910 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:46.931 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:46.931 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:46.952 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:46.952 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:46.957 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:46.959+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:46.973 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:46.973 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:47.005 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:47.005 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:47.020 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:24:47.058 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:47.061+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:47.159 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:47.163+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:47.183 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:47.183 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:47.197 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:24:47.260 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:47.265+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:47.313 INFO:tasks.rados.rados.0.smithi181.stdout:258: finishing copy_from racing read to smithi18128752-48 2024-04-26T01:24:47.359 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:47.359 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:47.361 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:47.363+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:47.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:24:47.414 INFO:tasks.rados.rados.0.smithi181.stdout:258: finishing copy_from to smithi18128752-48 2024-04-26T01:24:47.414 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 76 (ObjNum 97 snap 17 seq_num 97) dirty exists 2024-04-26T01:24:47.463 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:47.467+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:47.510 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:47.510 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:47.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:24:47.564 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:47.569+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:47.653 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:47.653 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:47.665 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:47.667+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:47.668 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:24:47.766 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:47.769+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:47.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:24:47.836 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:47.836 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:47.850 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:24:47.893 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T01:24:47.898 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:47.903+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:47.901 INFO:tasks.rados.rados.0.smithi181.stdout:258: done (2 left) 2024-04-26T01:24:47.901 INFO:tasks.rados.rados.0.smithi181.stdout:259: done (1 left) 2024-04-26T01:24:47.901 INFO:tasks.rados.rados.0.smithi181.stdout:260: done (0 left) 2024-04-26T01:24:47.902 INFO:tasks.rados.rados.0.smithi181.stdout:261: read oid 47 snap -1 2024-04-26T01:24:47.902 INFO:tasks.rados.rados.0.smithi181.stdout:261: expect (ObjNum 97 snap 17 seq_num 97) 2024-04-26T01:24:47.902 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 45 current snap is 21 2024-04-26T01:24:47.902 INFO:tasks.rados.rados.0.smithi181.stdout:262: seq_num 104 ranges {2113536=589824,2703360=540672,3244032=622592} 2024-04-26T01:24:47.916 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-26T01:24:47.929 INFO:tasks.rados.rados.0.smithi181.stdout:262: writing smithi18128752-45 from 2113536 to 2703360 tid 1 2024-04-26T01:24:47.935 INFO:tasks.rados.rados.0.smithi181.stdout:262: writing smithi18128752-45 from 2703360 to 3244032 tid 2 2024-04-26T01:24:47.943 INFO:tasks.rados.rados.0.smithi181.stdout:262: writing smithi18128752-45 from 3244032 to 3866624 tid 3 2024-04-26T01:24:47.943 INFO:tasks.rados.rados.0.smithi181.stdout:263: read oid 49 snap 19 2024-04-26T01:24:47.944 INFO:tasks.rados.rados.0.smithi181.stdout:263: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T01:24:47.944 INFO:tasks.rados.rados.0.smithi181.stdout:264: read oid 28 snap -1 2024-04-26T01:24:47.944 INFO:tasks.rados.rados.0.smithi181.stdout:264: expect (ObjNum 101 snap 17 seq_num 101) 2024-04-26T01:24:47.944 INFO:tasks.rados.rados.0.smithi181.stdout:265: read oid 46 snap 19 2024-04-26T01:24:47.944 INFO:tasks.rados.rados.0.smithi181.stdout:265: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:47.944 INFO:tasks.rados.rados.0.smithi181.stdout:266: snap_remove snap 20 2024-04-26T01:24:47.967 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:47.973+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:47.984 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:47.984 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:47.999 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:24:48.090 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:48.095+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:48.169 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:48.171+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:48.171 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:48.172 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:48.186 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:24:48.270 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:48.271+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:48.361 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:48.361 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:48.371 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:48.373+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:48.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:24:48.471 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:48.477+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:48.503 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:48.504 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:48.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:48.572 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:48.575+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:48.646 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:48.647 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:48.662 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:48.673 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:48.675+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:48.774 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:48.777+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:48.775 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-04-26T01:24:48.775 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T01:24:48.780 INFO:tasks.rados.rados.0.smithi181.stdout:266: done (5 left) 2024-04-26T01:24:48.780 INFO:tasks.rados.rados.0.smithi181.stdout:267: rollback oid 13 current snap is 21 2024-04-26T01:24:48.780 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 13 to 19 2024-04-26T01:24:48.781 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing write tid 1 to smithi18128752-45 2024-04-26T01:24:48.781 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing write tid 2 to smithi18128752-45 2024-04-26T01:24:48.781 INFO:tasks.rados.rados.0.smithi181.stdout:265: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:48.793 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:48.793 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:48.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:48.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-26T01:24:48.845 INFO:tasks.rados.rados.0.smithi181.stdout:264: expect (ObjNum 101 snap 17 seq_num 101) 2024-04-26T01:24:48.875 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:48.877+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:48.922 INFO:tasks.rados.rados.0.smithi181.stdout:261: expect (ObjNum 97 snap 17 seq_num 97) 2024-04-26T01:24:48.937 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:48.937 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:48.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:48.963 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing write tid 3 to smithi18128752-45 2024-04-26T01:24:48.963 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing write tid 4 to smithi18128752-45 2024-04-26T01:24:48.963 INFO:tasks.rados.rados.0.smithi181.stdout:263: expect (ObjNum 48 snap 0 seq_num 48) 2024-04-26T01:24:48.976 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:48.981+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing write tid 5 to smithi18128752-45 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:262: oid 45 updating version 0 to 101 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:262: oid 45 updating version 101 to 104 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:262: oid 45 version 104 is already newer than 103 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:262: oid 45 version 104 is already newer than 102 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 104 (ObjNum 104 snap 21 seq_num 104) dirty exists 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:262: left oid 45 (ObjNum 104 snap 21 seq_num 104) 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:261: done (5 left) 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:262: done (4 left) 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:263: done (3 left) 2024-04-26T01:24:48.984 INFO:tasks.rados.rados.0.smithi181.stdout:264: done (2 left) 2024-04-26T01:24:48.985 INFO:tasks.rados.rados.0.smithi181.stdout:265: done (1 left) 2024-04-26T01:24:48.985 INFO:tasks.rados.rados.0.smithi181.stdout:268: read oid 33 snap -1 2024-04-26T01:24:48.985 INFO:tasks.rados.rados.0.smithi181.stdout:268: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:48.985 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 16 current snap is 21 2024-04-26T01:24:48.985 INFO:tasks.rados.rados.0.smithi181.stdout:269: seq_num 105 ranges {4063232=606208,4669440=688128,5357568=475136} 2024-04-26T01:24:48.994 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:49.052 INFO:tasks.rados.rados.0.smithi181.stdout:269: writing smithi18128752-16 from 4063232 to 4669440 tid 1 2024-04-26T01:24:49.059 INFO:tasks.rados.rados.0.smithi181.stdout:269: writing smithi18128752-16 from 4669440 to 5357568 tid 2 2024-04-26T01:24:49.064 INFO:tasks.rados.rados.0.smithi181.stdout:269: writing smithi18128752-16 from 5357568 to 5832704 tid 3 2024-04-26T01:24:49.064 INFO:tasks.rados.rados.0.smithi181.stdout:270: setattr oid 19 current snap is 21 2024-04-26T01:24:49.064 INFO:tasks.rados.rados.0.smithi181.stdout:267: finishing rollback tid 0 to smithi18128752-13 2024-04-26T01:24:49.064 INFO:tasks.rados.rados.0.smithi181.stdout:267: finishing rollback tid 1 to smithi18128752-13 2024-04-26T01:24:49.064 INFO:tasks.rados.rados.0.smithi181.stdout:267: finishing rollback tid 2 to smithi18128752-13 2024-04-26T01:24:49.064 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 108 (ObjNum 91 snap 17 seq_num 91) dirty exists 2024-04-26T01:24:49.066 INFO:tasks.rados.rados.0.smithi181.stdout:267: done (3 left) 2024-04-26T01:24:49.066 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 12 current snap is 21 2024-04-26T01:24:49.066 INFO:tasks.rados.rados.0.smithi181.stdout:271: seq_num 106 ranges {1507328=622592,2129920=442368,2572288=524288,3096576=196608} 2024-04-26T01:24:49.077 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:49.079+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:49.081 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:49.081 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:49.082 INFO:tasks.rados.rados.0.smithi181.stdout:271: writing smithi18128752-12 from 1507328 to 2129920 tid 1 2024-04-26T01:24:49.085 INFO:tasks.rados.rados.0.smithi181.stdout:271: writing smithi18128752-12 from 2129920 to 2572288 tid 2 2024-04-26T01:24:49.088 INFO:tasks.rados.rados.0.smithi181.stdout:271: writing smithi18128752-12 from 2572288 to 3096576 tid 3 2024-04-26T01:24:49.090 INFO:tasks.rados.rados.0.smithi181.stdout:271: writing smithi18128752-12 from 3096576 to 3293184 tid 4 2024-04-26T01:24:49.090 INFO:tasks.rados.rados.0.smithi181.stdout:272: read oid 2 snap 17 2024-04-26T01:24:49.090 INFO:tasks.rados.rados.0.smithi181.stdout:272: expect deleted 2024-04-26T01:24:49.090 INFO:tasks.rados.rados.0.smithi181.stdout:273: snap_remove snap 18 2024-04-26T01:24:49.097 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:49.134 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:49.134 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:49.152 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:49.152 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:49.175 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:49.176 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:49.178 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:49.181+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:49.196 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:49.196 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:49.213 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:49.213 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:49.233 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:49.233 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:49.244 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:49.254 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:49.254 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:49.276 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:49.277 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:49.288 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:49.291+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:49.295 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:49.295 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:49.326 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:49.327 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:49.348 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:49.348 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:49.371 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:49.371 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:49.376 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:49.377 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:49.377 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:49.377 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:49.379 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:49.383+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:49.391 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:49.391 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:49.392 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:49.481 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:49.483+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:49.492 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:49.492 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:49.523 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:49.523 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:49.558 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:49.558 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:49.559 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:49.559 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:49.559 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:49.559 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:49.559 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:49.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:49.582 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:49.583+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:49.683 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:49.685+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:49.700 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:49.700 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:49.700 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:49.700 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:49.700 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.700 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:146 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.537708276000004, 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.841318758, 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 146 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.701 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165148+0000", 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6000000000000001e-06 2024-04-26T01:24:49.702 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165979+0000", 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00083094699999999996 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166013+0000", 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3386000000000002e-05 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166028+0000", 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5480000000000001e-05 2024-04-26T01:24:49.703 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189275+0000", 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.023246692999999999 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189585+0000", 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031012999999999999 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.190986+0000", 2024-04-26T01:24:49.704 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014009820000000001 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295619+0000", 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104633299 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005890+0000", 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71027072300000005 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.705 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006464+0000", 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057451799999999999 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:147 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.531468832999998, 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93684057600000004, 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:49.706 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 147 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:24:49.707 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171387+0000", 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4099999999999998e-06 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171984+0000", 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059681399999999998 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.708 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172016+0000", 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1779000000000002e-05 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172031+0000", 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5026e-05 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192109+0000", 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.020077514000000001 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.709 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192330+0000", 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00022101 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.193866+0000", 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001536239 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295801+0000", 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101934767 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.710 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.107290+0000", 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81148932200000001 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108226+0000", 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093569499999999995 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:148 3.es0 3:734f3d85:::smithi18128752-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:49.711 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.530794981, 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.137414704, 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 148 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.712 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172060+0000", 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.513e-06 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.713 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172069+0000", 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0930000000000003e-06 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172088+0000", 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9303000000000001e-05 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172103+0000", 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5586e-05 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.714 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496369+0000", 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32426532299999999 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496614+0000", 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024487400000000001 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.497656+0000", 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010425289999999999 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.715 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.599811+0000", 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10215491 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.308958+0000", 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70914692000000001 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.309474+0000", 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051565300000000003 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.716 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:149 3.es0 3:734f3d85:::smithi18128752-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.530714674000002, 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.54710463, 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:49.717 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 149 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.718 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172141+0000", 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.39e-06 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172148+0000", 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0600000000000002e-06 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172161+0000", 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3803e-05 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.719 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172171+0000", 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4069999999999999e-06 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498185+0000", 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.326014476 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498406+0000", 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000220461 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.720 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311032+0000", 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81262658200000004 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311366+0000", 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00033330099999999999 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.719244+0000", 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40787814999999999 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.721 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.243508331999998, 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0565503999999999, 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.722 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9999999999999998e-07 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.723 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459365+0000", 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8600000000000001e-05 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459378+0000", 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2989e-05 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459388+0000", 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0710000000000001e-05 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.724 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515199+0000", 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.055811056 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515896+0000", 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069664499999999997 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:49.725 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.241926817, 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54462607299999999, 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:24:49.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460929+0000", 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.72e-06 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460937+0000", 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.763e-06 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460965+0000", 2024-04-26T01:24:49.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8693000000000001e-05 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460979+0000", 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4072e-05 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005512+0000", 2024-04-26T01:24:49.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54453236000000005 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005553+0000", 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1464999999999997e-05 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:49.729 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.241311344000003, 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54510908400000002, 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:24:49.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461544+0000", 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461554+0000", 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3700000000000001e-06 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461582+0000", 2024-04-26T01:24:49.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8062e-05 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461596+0000", 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.432e-05 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006604+0000", 2024-04-26T01:24:49.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54500755300000003 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006652+0000", 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8041999999999998e-05 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.227433144999999, 2024-04-26T01:24:49.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0397411919999999, 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475422+0000", 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475446+0000", 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4428999999999999e-05 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475478+0000", 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1516000000000001e-05 2024-04-26T01:24:49.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.575602+0000", 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100124637 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515084+0000", 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93948142300000004 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515162+0000", 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8213999999999993e-05 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.203431921000004, 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60899775199999995, 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.737 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499424+0000", 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7630000000000001e-06 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499436+0000", 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.201e-05 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.739 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499465+0000", 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9142999999999999e-05 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499479+0000", 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4419e-05 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108343+0000", 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60886387600000003 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108420+0000", 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6540999999999998e-05 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=20, to_read={3:7f6c3760:::smithi18128752-23:head=0,4096,0}, subchunks={3:7f6c3760:::smithi18128752-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.101666467000001, 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91354819600000003, 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.741 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:49.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601264+0000", 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6395000000000003e-05 2024-04-26T01:24:49.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601293+0000", 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8829000000000001e-05 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514183+0000", 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91289044600000002 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.744 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514736+0000", 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055252600000000002 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.100653557000001, 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.113522984, 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.745 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602202+0000", 2024-04-26T01:24:49.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602223+0000", 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1036e-05 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602251+0000", 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7773e-05 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702368+0000", 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100117531 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715639+0000", 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0132701660000001 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715723+0000", 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4738000000000001e-05 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.000389255999998, 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81161888599999998, 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702467+0000", 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0370000000000001e-06 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702492+0000", 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5096000000000002e-05 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702519+0000", 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7152e-05 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702533+0000", 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4464000000000001e-05 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514000+0000", 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81146615499999997 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514084+0000", 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3981999999999997e-05 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:49.752 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 42.924853693999999, 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83750464400000002, 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778002+0000", 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.64e-06 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778026+0000", 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778058+0000", 2024-04-26T01:24:49.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2398999999999999e-05 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778076+0000", 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7408999999999999e-05 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615429+0000", 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83735304799999999 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615505+0000", 2024-04-26T01:24:49.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6057999999999996e-05 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=21, to_read={3:7e2e4071:::smithi18128752-29:head=0,4096,0}, subchunks={3:7e2e4071:::smithi18128752-29:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 42.797664456, 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018501300000003, 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905272+0000", 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2304000000000003e-05 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905302+0000", 2024-04-26T01:24:49.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9850000000000001e-05 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615005+0000", 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70970295900000002 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615374+0000", 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036989999999999999 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 42.796743857000003, 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60991441499999999, 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906112+0000", 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.176e-06 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906133+0000", 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1135999999999999e-05 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906161+0000", 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7783000000000001e-05 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906175+0000", 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3419e-05 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515955+0000", 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60978075899999995 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.516024+0000", 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9141999999999996e-05 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 42.696699582999997, 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71033649600000004, 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.763 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006156+0000", 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:24:49.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006182+0000", 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6262999999999999e-05 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006213+0000", 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0298999999999999e-05 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006233+0000", 2024-04-26T01:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0169e-05 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716419+0000", 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018600300000001 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716491+0000", 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2132000000000007e-05 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 42.696464730000002, 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71023971900000005, 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006391+0000", 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006420+0000", 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8932999999999999e-05 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006451+0000", 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1086e-05 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006469+0000", 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8016000000000002e-05 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716554+0000", 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71008492899999998 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716629+0000", 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4857999999999994e-05 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:254 3.es0 3:7f6c3760:::smithi18128752-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.505980301000001, 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71203804900000001, 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:49.770 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 254 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9299999999999997e-07 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196880+0000", 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2410000000000003e-06 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196890+0000", 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0142999999999999e-05 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.296984+0000", 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100093949 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303480+0000", 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.006496646 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303606+0000", 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012540699999999999 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706770+0000", 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40316384799999999 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706914+0000", 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000144666 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.908912+0000", 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20199735599999999 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:257 3.es0 3:7915a690:::smithi18128752-22:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.404325084999996, 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71064416799999996, 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.775 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 257 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298530+0000", 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7999999999999993e-07 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298537+0000", 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8700000000000003e-06 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298551+0000", 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4202999999999999e-05 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298559+0000", 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0730000000000005e-06 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709349+0000", 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41079035699999999 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709598+0000", 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024883099999999998 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.710488+0000", 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00088957200000000004 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.811226+0000", 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100738614 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.008990+0000", 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19776387300000001 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009173+0000", 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000182795 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:258 3.es0 3:7915a690:::smithi18128752-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:24:49.780 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.404283739, 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.99510092800000005, 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 258 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:49.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298571+0000", 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:49.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298585+0000", 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4046000000000001e-05 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298599+0000", 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4192999999999999e-05 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298608+0000", 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6030000000000005e-06 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712320+0000", 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41371240100000001 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712472+0000", 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000151586 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009806+0000", 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.297334565 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.010023+0000", 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000216698 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.293671+0000", 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28364788600000002 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:49.785 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:49.787 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:49.788 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:49.789+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:49.888 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:49.893+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:49.903 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T01:24:49.914 INFO:tasks.rados.rados.0.smithi181.stdout:273: done (5 left) 2024-04-26T01:24:49.914 INFO:tasks.rados.rados.0.smithi181.stdout:274: copy_from oid 45 from oid 36 current snap is 21 2024-04-26T01:24:49.914 INFO:tasks.rados.rados.0.smithi181.stdout:268: expect (ObjNum 32 snap 0 seq_num 32) 2024-04-26T01:24:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:49.920 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:49.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 111 (ObjNum 161 snap 0 seq_num 0) dirty exists 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:269: finishing write tid 1 to smithi18128752-16 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:269: finishing write tid 2 to smithi18128752-16 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:271: finishing write tid 1 to smithi18128752-12 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:271: finishing write tid 2 to smithi18128752-12 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:268: done (5 left) 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:270: done (4 left) 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:272: done (3 left) 2024-04-26T01:24:49.932 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 10 current snap is 21 2024-04-26T01:24:49.933 INFO:tasks.rados.rados.0.smithi181.stdout:271: finishing write tid 3 to smithi18128752-12 2024-04-26T01:24:49.933 INFO:tasks.rados.rados.0.smithi181.stdout:275: seq_num 107 ranges {901120=638976,1540096=458752,1998848=704512} 2024-04-26T01:24:49.935 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:49.948 INFO:tasks.rados.rados.0.smithi181.stdout:275: writing smithi18128752-10 from 901120 to 1540096 tid 1 2024-04-26T01:24:49.953 INFO:tasks.rados.rados.0.smithi181.stdout:275: writing smithi18128752-10 from 1540096 to 1998848 tid 2 2024-04-26T01:24:49.961 INFO:tasks.rados.rados.0.smithi181.stdout:275: writing smithi18128752-10 from 1998848 to 2703360 tid 3 2024-04-26T01:24:49.962 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 22 current snap is 21 2024-04-26T01:24:49.962 INFO:tasks.rados.rados.0.smithi181.stdout:271: finishing write tid 4 to smithi18128752-12 2024-04-26T01:24:49.962 INFO:tasks.rados.rados.0.smithi181.stdout:271: finishing write tid 5 to smithi18128752-12 2024-04-26T01:24:49.962 INFO:tasks.rados.rados.0.smithi181.stdout:276: seq_num 108 ranges {0=655360,655360=606208,1261568=475136,1736704=81920} 2024-04-26T01:24:49.968 INFO:tasks.rados.rados.0.smithi181.stdout:276: writing smithi18128752-22 from 0 to 655360 tid 1 2024-04-26T01:24:49.975 INFO:tasks.rados.rados.0.smithi181.stdout:276: writing smithi18128752-22 from 655360 to 1261568 tid 2 2024-04-26T01:24:49.980 INFO:tasks.rados.rados.0.smithi181.stdout:276: writing smithi18128752-22 from 1261568 to 1736704 tid 3 2024-04-26T01:24:49.982 INFO:tasks.rados.rados.0.smithi181.stdout:276: writing smithi18128752-22 from 1736704 to 1818624 tid 4 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:277: read oid 38 snap -1 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:277: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:278: rollback oid 41 current snap is 21 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 41 to 17 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: finishing write tid 3 to smithi18128752-16 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: finishing write tid 4 to smithi18128752-16 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: finishing write tid 5 to smithi18128752-16 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: oid 16 updating version 0 to 45 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: oid 16 updating version 45 to 48 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: oid 16 version 48 is already newer than 47 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: oid 16 version 48 is already newer than 46 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 48 (ObjNum 105 snap 21 seq_num 105) dirty exists 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:269: left oid 16 (ObjNum 105 snap 21 seq_num 105) 2024-04-26T01:24:49.983 INFO:tasks.rados.rados.0.smithi181.stdout:271: finishing write tid 6 to smithi18128752-12 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:271: oid 12 updating version 0 to 31 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:271: oid 12 updating version 31 to 32 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:271: oid 12 version 32 is already newer than 29 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:271: oid 12 updating version 32 to 33 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:271: oid 12 version 33 is already newer than 30 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 33 (ObjNum 106 snap 21 seq_num 106) dirty exists 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:271: left oid 12 (ObjNum 106 snap 21 seq_num 106) 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:269: done (6 left) 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:271: done (5 left) 2024-04-26T01:24:49.984 INFO:tasks.rados.rados.0.smithi181.stdout:279: snap_remove snap 19 2024-04-26T01:24:49.988 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:49.993+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:50.063 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:50.063 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:50.063 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:50.064 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:50.064 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:50.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:50.090 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:50.095+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:50.191 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:50.195+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 35/34 ECSubWrite(tid=5, reqid=client.4589.0:116, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 44.644518044999998, 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20106410399999999, 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.201 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559045+0000", 2024-04-26T01:24:50.202 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8929999999999999e-06 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559097+0000", 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2672e-05 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559128+0000", 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1019000000000002e-05 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.203 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659233+0000", 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100104779 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659393+0000", 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016010199999999999 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659420+0000", 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6333e-05 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.204 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760047+0000", 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100627094 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760107+0000", 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0211999999999999e-05 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.205 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.es5 40/34 ECSubWrite(tid=40, reqid=client.4589.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=40'23))", 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.194086830000003, 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30174757699999999, 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.206 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009476+0000", 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9800000000000001e-06 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009502+0000", 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6495999999999998e-05 2024-04-26T01:24:50.207 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009541+0000", 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8748999999999998e-05 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009564+0000", 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2773e-05 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.208 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109788+0000", 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100224176 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109803+0000", 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4487e-05 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311086+0000", 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201283252 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.209 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311222+0000", 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013566399999999999 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 41/34 ECSubWrite(tid=27, reqid=osd.0.0:27, at_version=41'17, trim_to=0'0, roll_forward_to=39'15))", 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.104113300999998, 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21182642600000001, 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.210 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:24:50.211 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7300000000000001e-07 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099485+0000", 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7493000000000002e-05 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099509+0000", 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.387e-05 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.212 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099527+0000", 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7598999999999999e-05 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110237+0000", 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.010710366000000001 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110253+0000", 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5319999999999999e-05 2024-04-26T01:24:50.213 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311234+0000", 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20098134600000001 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311274+0000", 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9959000000000001e-05 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.214 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=32, reqid=client.4589.0:266, at_version=41'17, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.028414769000001, 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13212590199999999, 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.215 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175146+0000", 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175147+0000", 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.13e-07 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175731+0000", 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000583973 2024-04-26T01:24:50.216 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175761+0000", 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0536e-05 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.175777+0000", 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6076e-05 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302772+0000", 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12699442499999999 2024-04-26T01:24:50.217 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302793+0000", 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1382999999999998e-05 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307218+0000", 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0044247139999999997 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.218 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307272+0000", 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4181999999999997e-05 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 41/30 ECSubWrite(tid=33, reqid=osd.5.0:33, at_version=41'19, trim_to=0'0, roll_forward_to=37'15))", 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.025442386999998, 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.129420758, 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.219 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178118+0000", 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.220 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178120+0000", 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178160+0000", 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9356000000000001e-05 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178190+0000", 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0179e-05 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.221 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.178206+0000", 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5809000000000001e-05 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.304772+0000", 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.12656614899999999 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.304787+0000", 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4893e-05 2024-04-26T01:24:50.222 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307500+0000", 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027131490000000002 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.307539+0000", 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9486000000000001e-05 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=68, reqid=client.4589.0:313, at_version=46'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:50.223 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 32.239661259000002, 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20854388500000001, 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:24:50.224 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963902+0000", 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964264+0000", 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036215900000000003 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.225 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964306+0000", 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1801999999999997e-05 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964334+0000", 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8133e-05 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.226 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968462+0000", 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041272710000000001 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968486+0000", 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4219999999999999e-05 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172363+0000", 2024-04-26T01:24:50.227 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20387691099999999 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172443+0000", 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0798999999999997e-05 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.228 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=69, reqid=client.4589.0:314, at_version=46'19, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 32.235498319000001, 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20440952700000001, 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.229 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968064+0000", 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9829999999999999e-06 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968174+0000", 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000109028 2024-04-26T01:24:50.230 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968220+0000", 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6106000000000002e-05 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968244+0000", 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4239e-05 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970652+0000", 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024076810000000001 2024-04-26T01:24:50.231 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970665+0000", 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3652000000000001e-05 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172450+0000", 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201785134 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172472+0000", 2024-04-26T01:24:50.232 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1704000000000001e-05 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=70, reqid=client.4589.0:315, at_version=46'20, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 32.232620162000003, 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201814353, 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.233 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970942+0000", 2024-04-26T01:24:50.234 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8339999999999999e-06 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971003+0000", 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0401e-05 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971040+0000", 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7469999999999999e-05 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.235 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971064+0000", 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3609e-05 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.171968+0000", 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200904202 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.171991+0000", 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2702e-05 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.236 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172718+0000", 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00072753100000000001 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172755+0000", 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6603999999999997e-05 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 56/44 ECSubWrite(tid=107, reqid=osd.14.0:107, at_version=56'36, trim_to=0'0, roll_forward_to=55'33))", 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:24:50.237 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.054024295000001, 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20195263099999999, 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:24:50.238 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149538+0000", 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149588+0000", 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0337000000000003e-05 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149623+0000", 2024-04-26T01:24:50.239 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4782e-05 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149641+0000", 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7886000000000001e-05 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.349947+0000", 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20030594800000001 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.240 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.349962+0000", 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5746000000000001e-05 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.351420+0000", 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014574519999999999 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.351489+0000", 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9412999999999994e-05 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.241 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 61/44 ECSubWrite(tid=127, reqid=client.4589.0:471, at_version=61'56, trim_to=0'0, roll_forward_to=59'47))", 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.972287+0000", 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.231273455, 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.17427974299999999, 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972287+0000", 2024-04-26T01:24:50.242 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972287+0000", 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972290+0000", 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.447e-06 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.243 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972359+0000", 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9303999999999999e-05 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972396+0000", 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6665999999999998e-05 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972420+0000", 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4392e-05 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.244 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.145794+0000", 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.173373588 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.145802+0000", 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6510000000000007e-06 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.146516+0000", 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00071390400000000001 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.245 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.146567+0000", 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0791000000000002e-05 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds6 61/44 ECSubWrite(tid=182, reqid=osd.4.0:178, at_version=0'0, trim_to=0'0, roll_forward_to=61'33))", 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 16.801144909000001, 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201923306, 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.246 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.247 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402417+0000", 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1299999999999998e-07 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402437+0000", 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0322e-05 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402460+0000", 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2972999999999998e-05 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.248 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402512+0000", 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2151999999999998e-05 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.603076+0000", 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20056403 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.603093+0000", 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6336000000000001e-05 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.249 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.604291+0000", 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001198253 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.604339+0000", 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8327e-05 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.250 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.ds6 65/63 ECSubRead(tid=3, to_read={3:b136579b:::smithi18128752-36:4=0,262144,0}, subchunks={3:b136579b:::smithi18128752-36:4=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:37.842193+0000", 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.361367921999999, 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201211578, 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842193+0000", 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.251 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842193+0000", 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842195+0000", 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.857e-06 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842212+0000", 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6993000000000001e-05 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.252 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842246+0000", 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4004999999999997e-05 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842264+0000", 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8320000000000001e-05 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.042660+0000", 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200396355 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.253 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.043404+0000", 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00074404799999999998 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.3s4 66/63 ECSubWrite(tid=254, reqid=client.4589.0:526, at_version=66'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 12.149238205, 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20384141, 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.254 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.255 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9299999999999998e-07 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054927+0000", 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060434499999999995 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054955+0000", 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7679e-05 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.256 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.155087+0000", 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100132031 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.255298+0000", 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100210753 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.255314+0000", 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5650000000000001e-05 2024-04-26T01:24:50.257 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.258078+0000", 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027649390000000001 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.258164+0000", 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5519999999999994e-05 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.258 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 67/44 ECSubWrite(tid=114, reqid=osd.5.0:114, at_version=67'80, trim_to=0'0, roll_forward_to=67'76))", 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.142024+0000", 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.061536712000001, 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.157131892, 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.142024+0000", 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.259 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.142024+0000", 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.142026+0000", 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.753e-06 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.142065+0000", 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9399e-05 2024-04-26T01:24:50.260 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.142095+0000", 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9896e-05 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.142130+0000", 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5292e-05 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.240106+0000", 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.097975096999999997 2024-04-26T01:24:50.261 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.240124+0000", 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8223000000000002e-05 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.299112+0000", 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.058988376000000002 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.299156+0000", 2024-04-26T01:24:50.262 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3856000000000002e-05 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s3 67/44 ECSubWrite(tid=262, reqid=osd.13.0:258, at_version=0'0, trim_to=0'0, roll_forward_to=67'43))", 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.233254+0000", 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.970306545, 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200779026, 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.263 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.233254+0000", 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.233254+0000", 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.233256+0000", 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9429999999999999e-06 2024-04-26T01:24:50.264 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.233278+0000", 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1535999999999998e-05 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.233308+0000", 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0035999999999998e-05 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.333448+0000", 2024-04-26T01:24:50.265 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100140431 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.333636+0000", 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000187672 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.333664+0000", 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8439e-05 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.266 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.433977+0000", 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10031313 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.434033+0000", 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5838999999999998e-05 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.7s1 67/30 ECSubWrite(tid=205, reqid=client.4589.0:529, at_version=0'0, trim_to=0'0, roll_forward_to=67'46))", 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.303370+0000", 2024-04-26T01:24:50.267 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.900190525999999, 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30256664599999999, 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303370+0000", 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303370+0000", 2024-04-26T01:24:50.268 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303372+0000", 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6169999999999999e-06 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303395+0000", 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.287e-05 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303419+0000", 2024-04-26T01:24:50.269 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4522e-05 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303433+0000", 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3987e-05 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.503760+0000", 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20032639099999999 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.270 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.503792+0000", 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2066000000000001e-05 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.605860+0000", 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102068162 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.605937+0000", 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7031000000000002e-05 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.271 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.7s1 67/30 ECSubWrite(tid=206, reqid=osd.4.0:206, at_version=67'48, trim_to=0'0, roll_forward_to=67'46))", 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.338051+0000", 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.865509831000001, 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26795410200000003, 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.272 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338051+0000", 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338051+0000", 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338053+0000", 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6530000000000001e-06 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.273 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338104+0000", 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0883000000000001e-05 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338139+0000", 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5799000000000001e-05 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438271+0000", 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10013135099999999 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.274 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.604310+0000", 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16603920699999999 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.604324+0000", 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3983e-05 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.605948+0000", 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001623902 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.275 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.606005+0000", 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7324000000000002e-05 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.3s4 67/63 ECSubWrite(tid=265, reqid=osd.13.0:265, at_version=67'7, trim_to=0'0, roll_forward_to=67'5))", 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.339927+0000", 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.863633889000001, 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.202186214, 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.276 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339927+0000", 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339927+0000", 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.277 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339929+0000", 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.81e-06 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339966+0000", 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7085999999999997e-05 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339995+0000", 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9581999999999999e-05 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.278 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.440127+0000", 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10013158799999999 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.540363+0000", 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10023565 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.540377+0000", 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4849e-05 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.279 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.542040+0000", 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001662771 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.542113+0000", 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2878000000000003e-05 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.7s1 67/30 ECSubWrite(tid=207, reqid=osd.4.0:207, at_version=67'50, trim_to=0'0, roll_forward_to=67'46))", 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.438320+0000", 2024-04-26T01:24:50.280 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.765240473, 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16834337399999999, 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438320+0000", 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438320+0000", 2024-04-26T01:24:50.281 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438321+0000", 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438380+0000", 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8816000000000001e-05 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.282 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438400+0000", 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0021999999999999e-05 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438413+0000", 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2403e-05 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.604907+0000", 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16649488900000001 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.283 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.604935+0000", 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7362999999999999e-05 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.606594+0000", 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016588499999999999 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.606664+0000", 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0057999999999999e-05 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.284 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 71/44 ECSubWrite(tid=190, reqid=client.4589.0:604, at_version=71'89, trim_to=0'0, roll_forward_to=71'87))", 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:43.811544+0000", 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.3920168579999999, 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13091910100000001, 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.811544+0000", 2024-04-26T01:24:50.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.811544+0000", 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.811546+0000", 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0829999999999998e-06 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.811636+0000", 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0158e-05 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.811668+0000", 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.188e-05 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.811690+0000", 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1458999999999999e-05 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.911918+0000", 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100228659 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.911949+0000", 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0496000000000001e-05 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.942247+0000", 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.030298148 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:43.942463+0000", 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021621800000000001 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.289 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:50.294 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:50.295 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:50.299+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:50.397 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:50.399+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:50.420 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:50.420 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:24:50.420 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:50.435 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:50.498 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:50.501+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:50.592 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:50.592 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:50.592 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:50.599 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:50.603+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:50.606 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:50.703 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:50.705+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:50.724 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:50.725 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:50.725 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:50.725 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:50.725 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.725 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/30 ECSubWrite(tid=47, reqid=client.4589.0:140, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:50.725 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:50.725 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 44.436213678999998, 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20188384300000001, 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.726 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292921+0000", 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8870000000000001e-06 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292964+0000", 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3152000000000003e-05 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.727 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.293004+0000", 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0065999999999999e-05 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.393103+0000", 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100099122 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493412+0000", 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10030921299999999 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.728 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493428+0000", 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494705+0000", 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012777590000000001 2024-04-26T01:24:50.729 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494803+0000", 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7477000000000002e-05 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 36/34 ECSubWrite(tid=26, reqid=client.4589.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.048838203000003, 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200889605, 2024-04-26T01:24:50.730 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.731 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680296+0000", 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8640000000000001e-06 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680320+0000", 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4202e-05 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680350+0000", 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9998999999999999e-05 2024-04-26T01:24:50.732 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780494+0000", 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100143549 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780762+0000", 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026773600000000003 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780776+0000", 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:24:50.733 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881147+0000", 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100370127 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881184+0000", 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7422000000000002e-05 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.734 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/30 ECSubWrite(tid=90, reqid=osd.4.0:90, at_version=41'10, trim_to=0'0, roll_forward_to=37'8))", 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.529479502000001, 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103318356, 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.735 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199654+0000", 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199704+0000", 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9656e-05 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.736 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199735+0000", 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1396000000000002e-05 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199762+0000", 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7092000000000001e-05 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.299984+0000", 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100221583 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.737 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.300000+0000", 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5906999999999999e-05 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302870+0000", 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028700879999999998 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302971+0000", 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010186799999999999 2024-04-26T01:24:50.738 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.327090372999997, 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201895835, 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.739 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402045+0000", 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2800000000000002e-06 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.740 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402074+0000", 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9366e-05 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402148+0000", 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4255e-05 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.741 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502254+0000", 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100106322 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502424+0000", 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000169259 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.742 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502449+0000", 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4909000000000002e-05 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603856+0000", 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101407649 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603938+0000", 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1794999999999999e-05 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.743 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 48/34 ECSubWrite(tid=49, reqid=client.4589.0:334, at_version=48'23, trim_to=0'0, roll_forward_to=45'21))", 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 31.364283370999999, 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103332259, 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.744 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364850+0000", 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0299999999999998e-07 2024-04-26T01:24:50.745 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365180+0000", 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032962699999999998 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365243+0000", 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3516000000000006e-05 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.746 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365268+0000", 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4522e-05 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365458+0000", 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000189946 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365484+0000", 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6800000000000001e-05 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.747 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.468084+0000", 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102600071 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.468182+0000", 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7073999999999999e-05 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 51/44 ECSubWrite(tid=206, reqid=client.4589.0:388, at_version=51'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:24:50.748 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.386188618999999, 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.138940597, 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.749 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342946+0000", 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343263+0000", 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000317664 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.750 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343300+0000", 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6293e-05 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343321+0000", 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1605999999999999e-05 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.751 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.479650+0000", 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.136328692 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.479665+0000", 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5393999999999999e-05 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.481807+0000", 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021412390000000001 2024-04-26T01:24:50.752 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.481885+0000", 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7959000000000004e-05 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 51/34 ECSubWrite(tid=59, reqid=client.4589.0:397, at_version=51'30, trim_to=0'0, roll_forward_to=49'28))", 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.175544260999999, 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10403118, 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.753 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.754 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553589+0000", 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6700000000000003e-07 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553663+0000", 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3734999999999998e-05 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553697+0000", 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4003e-05 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.755 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553722+0000", 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4678999999999999e-05 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.653970+0000", 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100247954 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.653997+0000", 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7089000000000001e-05 2024-04-26T01:24:50.756 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.657525+0000", 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0035286039999999999 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.657620+0000", 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4349000000000001e-05 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.757 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 51/34 ECSubWrite(tid=60, reqid=client.4589.0:397, at_version=0'0, trim_to=0'0, roll_forward_to=51'30))", 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.900282229999998, 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103627764, 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.758 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828851+0000", 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6000000000000004e-07 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828874+0000", 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3206000000000001e-05 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.759 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828896+0000", 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2209999999999999e-05 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828918+0000", 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1687000000000001e-05 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.929363+0000", 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100445169 2024-04-26T01:24:50.760 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.929394+0000", 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0456000000000001e-05 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932323+0000", 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029296819999999999 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932478+0000", 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015479400000000001 2024-04-26T01:24:50.761 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/30 ECSubWrite(tid=97, reqid=client.4589.0:400, at_version=52'20, trim_to=0'0, roll_forward_to=51'18))", 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 26.797099611, 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103051591, 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.762 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8599999999999999e-07 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.763 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932092+0000", 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8335999999999998e-05 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932116+0000", 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4176e-05 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932131+0000", 2024-04-26T01:24:50.764 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5187e-05 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932328+0000", 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019693900000000001 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932354+0000", 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5783e-05 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.765 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.035001+0000", 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102647556 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.035085+0000", 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3227999999999997e-05 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 53/34 ECSubWrite(tid=63, reqid=osd.11.0:63, at_version=53'34, trim_to=0'0, roll_forward_to=52'32))", 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:24.884726+0000", 2024-04-26T01:24:50.766 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 25.844406429999999, 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102783786, 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884726+0000", 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884726+0000", 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.767 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884727+0000", 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8999999999999996e-07 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884784+0000", 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7509000000000001e-05 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.768 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.884809+0000", 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4416000000000001e-05 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.984945+0000", 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100136112 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.985103+0000", 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015849399999999999 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.769 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.985133+0000", 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9413000000000001e-05 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.987420+0000", 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0022869169999999999 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.987510+0000", 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.0235000000000006e-05 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.770 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=84, reqid=osd.11.0:84, at_version=67'55, trim_to=0'0, roll_forward_to=66'51))", 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.799016+0000", 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.930116926, 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.203213587, 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799016+0000", 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.771 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799016+0000", 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799018+0000", 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9530000000000002e-06 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799071+0000", 2024-04-26T01:24:50.772 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2871999999999998e-05 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799101+0000", 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0176e-05 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799123+0000", 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2336000000000001e-05 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.000299+0000", 2024-04-26T01:24:50.773 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20117579299999999 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.000325+0000", 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5709999999999999e-05 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.002152+0000", 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001827953 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.774 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.002229+0000", 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6793999999999997e-05 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 67/44 ECSubWrite(tid=256, reqid=osd.13.0:256, at_version=67'41, trim_to=0'0, roll_forward_to=61'39))", 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.832797885, 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103034636, 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.775 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3e-07 2024-04-26T01:24:50.776 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896392+0000", 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6715000000000002e-05 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896424+0000", 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1458999999999998e-05 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896451+0000", 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7722999999999999e-05 2024-04-26T01:24:50.777 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.997106+0000", 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100655038 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.997139+0000", 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2872e-05 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.999288+0000", 2024-04-26T01:24:50.778 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021490179999999999 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.999369+0000", 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1081000000000005e-05 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 67/44 ECSubWrite(tid=258, reqid=osd.13.0:258, at_version=67'43, trim_to=0'0, roll_forward_to=61'39))", 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.899332+0000", 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.829800546, 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19960048699999999, 2024-04-26T01:24:50.779 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899332+0000", 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899332+0000", 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.780 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899334+0000", 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4300000000000001e-06 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899371+0000", 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7863e-05 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899413+0000", 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1721999999999998e-05 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.781 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899434+0000", 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0665999999999998e-05 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.097162+0000", 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19772809499999999 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.097188+0000", 2024-04-26T01:24:50.782 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6398999999999999e-05 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.098853+0000", 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016651839999999999 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.098933+0000", 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9128000000000005e-05 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.783 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=85, reqid=osd.11.0:84, at_version=0'0, trim_to=0'0, roll_forward_to=67'55))", 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.004160+0000", 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.724972234999999, 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30273594399999998, 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004160+0000", 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.784 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004160+0000", 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004162+0000", 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3969999999999999e-06 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004181+0000", 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9018999999999999e-05 2024-04-26T01:24:50.785 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004205+0000", 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4553e-05 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004221+0000", 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5282999999999999e-05 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.104433+0000", 2024-04-26T01:24:50.786 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10021221199999999 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.104464+0000", 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1556000000000001e-05 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306840+0000", 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.202375208 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.787 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306896+0000", 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6715999999999997e-05 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=86, reqid=osd.11.0:86, at_version=67'57, trim_to=0'0, roll_forward_to=67'55))", 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.006383+0000", 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.722749832, 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30055294300000002, 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.788 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006383+0000", 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006383+0000", 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006385+0000", 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:24:50.789 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006423+0000", 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8872000000000003e-05 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006453+0000", 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9529000000000001e-05 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006469+0000", 2024-04-26T01:24:50.790 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5982999999999999e-05 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.205542+0000", 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.199073163 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.205561+0000", 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8777e-05 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.791 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306904+0000", 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101343403 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306936+0000", 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1466e-05 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=87, reqid=osd.11.0:87, at_version=67'59, trim_to=0'0, roll_forward_to=67'55))", 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.007888+0000", 2024-04-26T01:24:50.792 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.721244823999999, 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30151905099999998, 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007888+0000", 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007888+0000", 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.793 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007890+0000", 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007928+0000", 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8844999999999998e-05 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007958+0000", 2024-04-26T01:24:50.794 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9408999999999999e-05 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007973+0000", 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5583999999999999e-05 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306254+0000", 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29828028099999998 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.795 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306291+0000", 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6989000000000001e-05 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.309343+0000", 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003052758 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.309407+0000", 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3435000000000005e-05 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.796 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds3 67/63 ECSubWrite(tid=7, reqid=osd.7.0:7, at_version=67'35, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.202408+0000", 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.526724230999999, 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10303036, 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.202408+0000", 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.797 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.202408+0000", 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.202409+0000", 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9399999999999993e-07 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.798 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.202464+0000", 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4265000000000003e-05 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.202495+0000", 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1715999999999999e-05 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.202519+0000", 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3782e-05 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.799 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.300923+0000", 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.098403355999999997 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.300938+0000", 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5823000000000001e-05 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.305329+0000", 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0043910210000000002 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.800 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.305439+0000", 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000109403 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 67/30 ECSubWrite(tid=154, reqid=osd.14.0:154, at_version=67'35, trim_to=0'0, roll_forward_to=67'33))", 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.295910+0000", 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.433222339, 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20202020500000001, 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.801 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295910+0000", 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295910+0000", 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295911+0000", 2024-04-26T01:24:50.802 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0070000000000001e-06 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295981+0000", 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9771000000000007e-05 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.296009+0000", 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8223000000000001e-05 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.396170+0000", 2024-04-26T01:24:50.803 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10016037699999999 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.496393+0000", 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.1002237 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.496413+0000", 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9202999999999999e-05 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.804 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.497857+0000", 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001444503 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.497930+0000", 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3421000000000001e-05 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 67/30 ECSubWrite(tid=155, reqid=osd.14.0:154, at_version=0'0, trim_to=0'0, roll_forward_to=67'35))", 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.929297042, 2024-04-26T01:24:50.805 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200835817, 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.806 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4300000000000002e-07 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799857+0000", 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0562999999999999e-05 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799879+0000", 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1872999999999999e-05 2024-04-26T01:24:50.807 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799900+0000", 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1682e-05 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.834641+0000", 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.034740756999999997 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.834670+0000", 2024-04-26T01:24:50.808 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8345999999999999e-05 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:40.000546+0000", 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16587683 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:40.000671+0000", 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012502300000000001 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.809 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 78/30 ECSubWrite(tid=221, reqid=client.4589.0:679, at_version=78'56, trim_to=0'0, roll_forward_to=72'53))", 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.032450+0000", 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.696683002, 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10285335399999999, 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.032450+0000", 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.810 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.032450+0000", 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.032450+0000", 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0400000000000002e-07 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.032588+0000", 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000137892 2024-04-26T01:24:50.811 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.032603+0000", 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5390999999999999e-05 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.032615+0000", 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2116999999999999e-05 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.133209+0000", 2024-04-26T01:24:50.812 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100593869 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.133227+0000", 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8019000000000001e-05 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.135258+0000", 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0020302979999999998 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.135303+0000", 2024-04-26T01:24:50.813 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.5364000000000001e-05 2024-04-26T01:24:50.814 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.814 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.814 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.814 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.814 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.814 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:50.815 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:24:50.816 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:50.821+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:50.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:50.906 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.5 to 1 (10000) 2024-04-26T01:24:50.912 INFO:tasks.rados.rados.0.smithi181.stdout:279: done (5 left) 2024-04-26T01:24:50.912 INFO:tasks.rados.rados.0.smithi181.stdout:280: snap_create 2024-04-26T01:24:50.912 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 1 to smithi18128752-10 2024-04-26T01:24:50.912 INFO:tasks.rados.rados.0.smithi181.stdout:277: expect (ObjNum 50 snap 0 seq_num 50) 2024-04-26T01:24:50.916 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:50.921+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:50.931 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-26T01:24:50.950 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:50.950 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:50.950 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.950 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.9s0 78/30 ECSubRead(tid=230, to_read={3:93dca175:::smithi18128752-10:head=0,4096,0}, subchunks={3:93dca175:::smithi18128752-10:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:50.950 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.766434+0000", 2024-04-26T01:24:50.950 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.18834114399999999, 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18838918700000001, 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766434+0000", 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.951 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766434+0000", 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766434+0000", 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766434+0000", 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.952 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766452+0000", 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7278e-05 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766459+0000", 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5510000000000002e-06 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.953 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 78/30 ECSubWriteReply(tid=227, last_complete=78'59, committed=1, applied=1))", 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.766858+0000", 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.18791782300000001, 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18805443399999999, 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.954 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766858+0000", 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766858+0000", 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766859+0000", 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1230000000000001e-06 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.955 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766867+0000", 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9529999999999999e-06 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766889+0000", 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2042999999999999e-05 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766908+0000", 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8859999999999999e-05 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.956 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.9s0 78/30 ECSubReadReply(tid=230, attrs_read=0))", 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.767420+0000", 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.18735565200000001, 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.187571511, 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767420+0000", 2024-04-26T01:24:50.957 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767420+0000", 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767421+0000", 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6840000000000001e-06 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.958 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767442+0000", 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0222999999999999e-05 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767468+0000", 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6205999999999999e-05 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767481+0000", 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3589e-05 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.959 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.9s0 78/30 ECSubReadReply(tid=230, attrs_read=0))", 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.767837+0000", 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.18693797700000001, 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.18722929099999999, 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767837+0000", 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.960 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767837+0000", 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767839+0000", 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1340000000000001e-06 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767854+0000", 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5421999999999999e-05 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.961 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767872+0000", 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.823e-05 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.767888+0000", 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5727e-05 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.962 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:683 3.9s0 3:93dca175:::smithi18128752-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.89347785099999999, 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.89385662300000002, 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 683 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.963 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.964 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7300000000000005e-07 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061302+0000", 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5860000000000001e-06 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061313+0000", 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0689999999999999e-05 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.965 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161394+0000", 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100081715 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161974+0000", 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057978700000000001 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.162126+0000", 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015176100000000001 2024-04-26T01:24:50.966 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.765908+0000", 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60378186599999994 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766058+0000", 2024-04-26T01:24:50.967 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015031300000000001 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 78/44 ECSubWriteReply(tid=229, last_complete=78'118, committed=1, applied=1))", 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.869263+0000", 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.085511960999999997, 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.086023394000000003, 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.968 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.869263+0000", 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.869263+0000", 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.969 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.869264+0000", 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7100000000000002e-07 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.869268+0000", 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8750000000000002e-06 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.869280+0000", 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1972e-05 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.970 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:677 3.0s0 3:01bb5e34:::smithi18128752-45:head [copy-from ver 14 in=55b] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.0344503780000001, 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0350411859999999, 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 677 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:50.971 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920326+0000", 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.432e-06 2024-04-26T01:24:50.972 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920334+0000", 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3699999999999997e-06 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920349+0000", 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5540999999999999e-05 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920360+0000", 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0868e-05 2024-04-26T01:24:50.973 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920475+0000", 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011452599999999999 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920488+0000", 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3478999999999999e-05 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.923555+0000", 2024-04-26T01:24:50.974 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030671539999999999 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.923806+0000", 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025074800000000002 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.745394+0000", 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.82158756200000005 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.748239+0000", 2024-04-26T01:24:50.975 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028450989999999998 2024-04-26T01:24:50.976 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.976 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:50.976 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.976 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:50.976 INFO:teuthology.orchestra.run.smithi071.stdout: ], 2024-04-26T01:24:50.976 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 7 2024-04-26T01:24:50.976 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:50.977 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:276: finishing write tid 1 to smithi18128752-22 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:278: finishing rollback tid 0 to smithi18128752-41 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:278: finishing rollback tid 1 to smithi18128752-41 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 2 to smithi18128752-10 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 3 to smithi18128752-10 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 4 to smithi18128752-10 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:278: finishing rollback tid 2 to smithi18128752-41 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 41 v 79 (ObjNum 85 snap 16 seq_num 85) dirty exists 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:274: finishing copy_from racing read to smithi18128752-45 2024-04-26T01:24:50.980 INFO:tasks.rados.rados.0.smithi181.stdout:276: finishing write tid 2 to smithi18128752-22 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:276: finishing write tid 3 to smithi18128752-22 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:276: finishing write tid 4 to smithi18128752-22 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:276: finishing write tid 5 to smithi18128752-22 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 5 to smithi18128752-10 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 10 updating version 0 to 57 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 10 updating version 57 to 58 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 10 updating version 58 to 59 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 10 version 59 is already newer than 56 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 59 (ObjNum 107 snap 21 seq_num 107) dirty exists 2024-04-26T01:24:50.981 INFO:tasks.rados.rados.0.smithi181.stdout:275: left oid 10 (ObjNum 107 snap 21 seq_num 107) 2024-04-26T01:24:51.024 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:51.027+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:51.064 INFO:tasks.rados.rados.0.smithi181.stdout:274: finishing copy_from to smithi18128752-45 2024-04-26T01:24:51.064 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 14 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-04-26T01:24:51.100 INFO:tasks.rados.rados.0.smithi181.stdout:276: finishing write tid 6 to smithi18128752-22 2024-04-26T01:24:51.100 INFO:tasks.rados.rados.0.smithi181.stdout:276: oid 22 updating version 0 to 64 2024-04-26T01:24:51.100 INFO:tasks.rados.rados.0.smithi181.stdout:276: oid 22 updating version 64 to 66 2024-04-26T01:24:51.101 INFO:tasks.rados.rados.0.smithi181.stdout:276: oid 22 version 66 is already newer than 65 2024-04-26T01:24:51.101 INFO:tasks.rados.rados.0.smithi181.stdout:276: oid 22 version 66 is already newer than 62 2024-04-26T01:24:51.101 INFO:tasks.rados.rados.0.smithi181.stdout:276: oid 22 version 66 is already newer than 63 2024-04-26T01:24:51.101 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 66 (ObjNum 108 snap 21 seq_num 108) dirty exists 2024-04-26T01:24:51.101 INFO:tasks.rados.rados.0.smithi181.stdout:276: left oid 22 (ObjNum 108 snap 21 seq_num 108) 2024-04-26T01:24:51.103 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:51.103 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:24:51.103 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:24:51.103 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:24:51.103 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:51.118 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:51.119 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:51.125+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:51.221 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:51.223+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:24:51.284 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:154 3.9s0 3:958f0a11:::smithi18128752-2:head [copy-from ver 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 45.011426921000002, 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59153377200000001, 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 154 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.285 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274163+0000", 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.286 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274166+0000", 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.027e-06 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274180+0000", 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3723000000000001e-05 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.287 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274213+0000", 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3698999999999997e-05 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274229+0000", 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5928999999999999e-05 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.288 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274415+0000", 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018537499999999999 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.274584+0000", 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016943000000000001 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.662522+0000", 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.38793827400000003 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.289 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.664271+0000", 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00174884 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.865553+0000", 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20128183599999999 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.865697+0000", 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014363900000000001 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.290 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:196 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.208190149000004, 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61790984400000004, 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 196 2024-04-26T01:24:51.291 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.292 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8169999999999999e-06 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077416+0000", 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3783e-05 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7209000000000002e-05 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.293 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4168999999999998e-05 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077630+0000", 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015196599999999999 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077649+0000", 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9262999999999998e-05 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.294 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.181790+0000", 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10414122100000001 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182072+0000", 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00028194299999999999 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.183846+0000", 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001774184 2024-04-26T01:24:51.295 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189006+0000", 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0051592809999999999 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695163+0000", 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50615769499999996 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695310+0000", 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014631299999999999 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.296 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:197 3.9s0 3:93dca175:::smithi18128752-10:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.208061729000001, 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61806578000000001, 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.297 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 197 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.298 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077529+0000", 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3239999999999999e-06 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077537+0000", 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0560000000000005e-06 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077556+0000", 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8456000000000001e-05 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.299 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077573+0000", 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6566e-05 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077807+0000", 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023464800000000001 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077822+0000", 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5206e-05 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.300 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184696+0000", 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10687403099999999 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184837+0000", 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000141113 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.186190+0000", 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001352575 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.301 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189008+0000", 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028181450000000002 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695495+0000", 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50648651300000003 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695594+0000", 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9147000000000005e-05 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.302 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:198 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.207995543999999, 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61826488800000001, 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 198 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.303 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077595+0000", 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.167e-06 2024-04-26T01:24:51.304 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077601+0000", 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6500000000000001e-06 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077619+0000", 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7569000000000001e-05 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077643+0000", 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4652999999999999e-05 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.305 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077989+0000", 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034534799999999999 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078006+0000", 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7125999999999998e-05 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187076+0000", 2024-04-26T01:24:51.306 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10906990799999999 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187274+0000", 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000198221 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.188283+0000", 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001008994 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189285+0000", 2024-04-26T01:24:51.307 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001002221 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695760+0000", 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.506474914 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695859+0000", 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9117000000000007e-05 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.308 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:315 3.0s0 3:03a7f525:::smithi18128752-19:head [setxattr _header (56) in=7b] snapc 6=[4] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 33.570079202999999, 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.53997301399999997, 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 315 2024-04-26T01:24:51.309 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.310 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715511+0000", 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7000000000000004e-07 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715515+0000", 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0879999999999997e-06 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715523+0000", 2024-04-26T01:24:51.311 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.605e-06 2024-04-26T01:24:51.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.316 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.316 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715529+0000", 2024-04-26T01:24:51.316 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1770000000000003e-06 2024-04-26T01:24:51.316 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.316 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855140+0000", 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13961058700000001 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855264+0000", 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012369900000000001 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969747+0000", 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.114483211 2024-04-26T01:24:51.317 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.969923+0000", 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000175954 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970982+0000", 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010588049999999999 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.318 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972408+0000", 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014263259999999999 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.254671+0000", 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28226294200000002 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.255484+0000", 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00081285000000000005 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.319 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:316 3.0s0 3:03a7f525:::smithi18128752-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 33.570052677, 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86208679600000004, 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 316 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.320 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715538+0000", 2024-04-26T01:24:51.321 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.68e-07 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715540+0000", 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.413e-06 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715547+0000", 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.131e-06 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.322 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715553+0000", 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6640000000000003e-06 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855355+0000", 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13980195500000001 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855595+0000", 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000240265 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.323 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971741+0000", 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.116145744 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972105+0000", 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036422899999999999 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272430+0000", 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30032516799999998 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.324 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272617+0000", 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018708799999999999 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.577624+0000", 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305006571 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:342 3.0s0 3:025aaed1:::smithi18128752-27:head [append 0~0] snapc 7=[4] ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-04-26T01:24:51.325 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:19.288497+0000", 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 31.997092452, 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58144454300000004, 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 342 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288497+0000", 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288497+0000", 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288502+0000", 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.2699999999999998e-06 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.327 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288518+0000", 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6220000000000001e-05 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288564+0000", 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6205999999999998e-05 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.288589+0000", 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.525e-05 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.328 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.363623+0000", 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.075033889000000006 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.363858+0000", 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023478600000000001 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.366695+0000", 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028370309999999998 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.329 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.372219+0000", 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0055238170000000003 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.869341+0000", 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49712220899999998 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.869942+0000", 2024-04-26T01:24:51.330 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060086500000000001 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(osd.4.21:9 3.0s0 3:0c3f65fa:::smithi18128752-8:head [assert-version v29,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 22.402368423999999, 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79000656700000005, 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:51.331 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883225+0000", 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2959999999999999e-06 2024-04-26T01:24:51.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883240+0000", 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4977000000000001e-05 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883273+0000", 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3818e-05 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883296+0000", 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2282000000000001e-05 2024-04-26T01:24:51.333 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.368761+0000", 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48546481800000002 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.368922+0000", 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000161039 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.334 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.673228+0000", 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30430633699999998 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:468 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~589824] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.858294+0000", 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.427296081000001, 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59054596400000003, 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.335 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 468 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.858294+0000", 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.858294+0000", 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.336 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.858297+0000", 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3000000000000002e-06 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.859432+0000", 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001134475 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.859461+0000", 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9132000000000002e-05 2024-04-26T01:24:51.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.859477+0000", 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5956000000000001e-05 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.941451+0000", 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.081974285999999993 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.338 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.941775+0000", 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032373599999999998 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.944167+0000", 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0023926170000000001 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968585+0000", 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.024417726000000001 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.339 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.448719+0000", 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48013421499999998 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.448840+0000", 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012052099999999999 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:469 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~573440] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.867391+0000", 2024-04-26T01:24:51.340 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.418198475000001, 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.58168562499999998, 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 469 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.341 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.867391+0000", 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.342 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.867391+0000", 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.867394+0000", 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3970000000000001e-06 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.868151+0000", 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00075715999999999999 2024-04-26T01:24:51.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.868183+0000", 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1742999999999997e-05 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.868198+0000", 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5773e-05 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.946903+0000", 2024-04-26T01:24:51.344 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.078704791999999996 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.947177+0000", 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027333499999999998 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.948983+0000", 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018060680000000001 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.345 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968588+0000", 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.019605043999999999 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449003+0000", 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48041582199999999 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449077+0000", 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3491000000000005e-05 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.346 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:470 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~196608] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870374+0000", 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.415216315999999, 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57888931700000001, 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 470 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.347 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870374+0000", 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870374+0000", 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870376+0000", 2024-04-26T01:24:51.348 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.413e-06 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870658+0000", 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000282188 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870690+0000", 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2169000000000003e-05 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870706+0000", 2024-04-26T01:24:51.349 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5739999999999998e-05 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968816+0000", 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.098110003000000001 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.968990+0000", 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017443100000000001 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.350 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.970055+0000", 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010643009999999999 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.973981+0000", 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0039264759999999999 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449202+0000", 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.475220637 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.351 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.449263+0000", 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0958999999999997e-05 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:471 3.0s0 3:01bb5e34:::smithi18128752-45:head [setxattr _header (57) in=7b] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.414856883999999, 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67904865400000003, 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.352 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 471 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.353 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870734+0000", 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.124e-06 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870740+0000", 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4530000000000001e-06 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.354 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870751+0000", 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1126e-05 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870759+0000", 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4370000000000007e-06 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.971074+0000", 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100315001 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.355 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.971323+0000", 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024919900000000002 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972178+0000", 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00085481999999999995 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.976513+0000", 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.004335317 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.356 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.549610+0000", 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57309677199999998 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.549782+0000", 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017140499999999999 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:472 3.0s0 3:01bb5e34:::smithi18128752-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:24:51.357 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 18.414818525000001, 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83389453400000002, 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 472 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.358 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870772+0000", 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3699999999999999e-07 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.359 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870776+0000", 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.416e-06 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870786+0000", 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0652999999999999e-05 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870795+0000", 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4640000000000006e-06 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.360 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.974442+0000", 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10364701900000001 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.974615+0000", 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017313100000000001 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.550708+0000", 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57609289699999999 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.361 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.550827+0000", 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000119185 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.704666+0000", 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15383893200000001 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.362 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:489 3.0s0 3:01bb5e34:::smithi18128752-45:head [read 0~2113536,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:34.787606+0000", 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 16.497984236000001, 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61237686899999999, 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 489 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.363 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787606+0000", 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787606+0000", 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787609+0000", 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.286e-06 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.364 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787624+0000", 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5383000000000002e-05 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787663+0000", 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8411999999999997e-05 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787695+0000", 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2780000000000001e-05 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.365 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787811+0000", 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000115412 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787816+0000", 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5740000000000003e-06 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.945236+0000", 2024-04-26T01:24:51.366 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.157420054 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.945342+0000", 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000105466 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:35.399982+0000", 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.454640502 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:490 3.0s0 3:01bb5e34:::smithi18128752-45:head [read 0~2113536,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:34.787750+0000", 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 16.497839426999999, 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61467409500000003, 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 490 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.368 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787750+0000", 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787750+0000", 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787752+0000", 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.463e-06 2024-04-26T01:24:51.369 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787760+0000", 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8769999999999993e-06 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787776+0000", 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6011999999999998e-05 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787792+0000", 2024-04-26T01:24:51.370 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6597000000000001e-05 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787923+0000", 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000130739 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.787927+0000", 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.732e-06 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.371 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.945891+0000", 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15796459400000001 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:34.946023+0000", 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013123099999999999 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:35.402424+0000", 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.45640185 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.372 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:677 3.0s0 3:01bb5e34:::smithi18128752-45:head [copy-from ver 14 in=55b] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.365264821, 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.149832408, 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 677 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.373 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920326+0000", 2024-04-26T01:24:51.374 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.432e-06 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920334+0000", 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3699999999999997e-06 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920349+0000", 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5540999999999999e-05 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.375 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920360+0000", 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0868e-05 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920475+0000", 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011452599999999999 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920488+0000", 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3478999999999999e-05 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.376 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.923555+0000", 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030671539999999999 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.923806+0000", 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025074800000000002 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.745394+0000", 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.82158756200000005 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.377 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.748239+0000", 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028450989999999998 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.069994+0000", 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32175474900000001 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.070157+0000", 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016388000000000001 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.378 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:678 3.0s0 3:01bb5e34:::smithi18128752-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.3652111740000001, 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.82652194000000001, 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.379 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 678 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.380 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920382+0000", 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9299999999999999e-06 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920390+0000", 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2300000000000008e-06 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.381 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920397+0000", 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6080000000000001e-06 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920586+0000", 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018897900000000001 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920597+0000", 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1467e-05 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.382 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.924206+0000", 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036080420000000001 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.924340+0000", 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000134797 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.746704+0000", 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.82236340299999999 2024-04-26T01:24:51.383 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.746807+0000", 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103463 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.746901+0000", 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3420999999999999e-05 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.384 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:680 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~458752] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:49.959707+0000", 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.325882797, 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60529759900000002, 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 680 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.385 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.959707+0000", 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.959707+0000", 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.959709+0000", 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.48e-06 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.386 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.960131+0000", 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00042201800000000002 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.960143+0000", 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2815e-05 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.060274+0000", 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100130651 2024-04-26T01:24:51.387 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.060362+0000", 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.8106e-05 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.060503+0000", 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000140727 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061485+0000", 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00098211000000000001 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161518+0000", 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10003340099999999 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.564904+0000", 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40338564300000002 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.389 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.565005+0000", 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000100648 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:682 3.9s0 3:93dca175:::smithi18128752-10:head [setxattr _header (57) in=7b] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.061240+0000", 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.224350286, 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.603996635, 2024-04-26T01:24:51.390 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 682 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061240+0000", 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.391 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061240+0000", 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061241+0000", 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2899999999999999e-06 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061248+0000", 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5429999999999996e-06 2024-04-26T01:24:51.392 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061269+0000", 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0665999999999998e-05 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061283+0000", 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4433000000000001e-05 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.064193+0000", 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029096119999999998 2024-04-26T01:24:51.393 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.064293+0000", 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9926000000000005e-05 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.064738+0000", 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00044460600000000001 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161532+0000", 2024-04-26T01:24:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.096794790000000006 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.665126+0000", 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.503593348 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.665236+0000", 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011042100000000001 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.395 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:683 3.9s0 3:93dca175:::smithi18128752-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.2242922940000001, 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90844643700000005, 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 683 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.396 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7300000000000005e-07 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.397 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061302+0000", 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5860000000000001e-06 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061313+0000", 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0689999999999999e-05 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161394+0000", 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100081715 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.398 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161974+0000", 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057978700000000001 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.162126+0000", 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015176100000000001 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.765908+0000", 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60378186599999994 2024-04-26T01:24:51.399 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766058+0000", 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015031300000000001 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.969744+0000", 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.203685846 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:24:51.400 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:24:51.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:51.403 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:51.405+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:51.503 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:51.507+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:51.531 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:51.531 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:51.531 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:51.532 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:51.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:51.585 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:51.631 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:51.635+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:51.667 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:51.667 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:51.667 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:24:51.667 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:24:51.668 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:51.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:51.704 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:51.707+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:51.745 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:51.746 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:51.766 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:51.767 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:51.787 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:51.787 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:51.801 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:51.801 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:24:51.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:24:51.801 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:24:51.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.801 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:169 3.as0 3:56e8b2fd:::smithi18128752-12:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:51.801 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.506020536000001, 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.415445014, 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 169 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297980+0000", 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3359999999999997e-06 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297998+0000", 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8499999999999999e-05 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298041+0000", 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3171999999999999e-05 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398202+0000", 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10016098800000001 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398693+0000", 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049036800000000003 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398913+0000", 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220051 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.610194+0000", 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.211280998 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612042+0000", 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018484860000000001 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713242+0000", 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10119953 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713420+0000", 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017858500000000001 2024-04-26T01:24:51.806 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:170 3.as0 3:56e8b2fd:::smithi18128752-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.405755753999998, 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.213653854, 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.807 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 170 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:24:51.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398242+0000", 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0530000000000001e-06 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398252+0000", 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4830000000000005e-06 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398271+0000", 2024-04-26T01:24:51.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-05 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398282+0000", 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1476e-05 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404421+0000", 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006138592 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:51.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404504+0000", 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2750999999999994e-05 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611404+0000", 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20690029200000001 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611712+0000", 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030829799999999998 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611894+0000", 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018167899999999999 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:212 3.as0 3:56e8b2fd:::smithi18128752-12:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 43.682997104999998, 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.306234695, 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:51.812 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:24:51.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121002+0000", 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.366e-06 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121071+0000", 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9202000000000006e-05 2024-04-26T01:24:51.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121106+0000", 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5052999999999998e-05 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121125+0000", 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8219e-05 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221415+0000", 2024-04-26T01:24:51.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029012800000001 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221427+0000", 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.187e-05 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222099+0000", 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067277899999999998 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222322+0000", 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000222272 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.224059+0000", 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017374059999999999 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226116+0000", 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020565679999999999 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426988+0000", 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20087223500000001 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.427234+0000", 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024559700000000001 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:51.818 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.578242142000001, 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28191508700000001, 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225756+0000", 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.756e-06 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225765+0000", 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225784+0000", 2024-04-26T01:24:51.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.325876+0000", 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100092051 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507611+0000", 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.181735758 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507669+0000", 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7361999999999997e-05 2024-04-26T01:24:51.821 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.401179593000002, 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20097827700000001, 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402816+0000", 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402817+0000", 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.026e-06 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402832+0000", 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4684999999999999e-05 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402851+0000", 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9338e-05 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.402863+0000", 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1832e-05 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.503005+0000", 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014197399999999 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.503027+0000", 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1903999999999999e-05 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.603709+0000", 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10068225 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.603795+0000", 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5267999999999997e-05 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 41/30 ECSubWrite(tid=94, reqid=osd.4.0:94, at_version=41'18, trim_to=0'0, roll_forward_to=39'14))", 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.299933563000003, 2024-04-26T01:24:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202154895, 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:24:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504063+0000", 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4900000000000001e-07 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504085+0000", 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2342999999999999e-05 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504093+0000", 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504100+0000", 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2470000000000002e-06 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704414+0000", 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20031389399999999 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704439+0000", 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.499e-05 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706155+0000", 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017162480000000001 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706217+0000", 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1878999999999995e-05 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s2 52/44 ECSubWrite(tid=213, reqid=osd.13.0:212, at_version=0'0, trim_to=0'0, roll_forward_to=52'22))", 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 27.446325290000001, 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30091122399999998, 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357672+0000", 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.75e-06 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357708+0000", 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5781999999999997e-05 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357774+0000", 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6131999999999996e-05 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.457877+0000", 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100102935 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458051+0000", 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017357000000000001 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458066+0000", 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5526e-05 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658511+0000", 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044448300000001 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658582+0000", 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1045999999999997e-05 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.as0 65/63 ECSubRead(tid=24, to_read={3:56e8b2fd:::smithi18128752-12:head=0,4096,0}, subchunks={3:56e8b2fd:::smithi18128752-12:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.859187379, 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20271442200000001, 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944842+0000", 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3609000000000003e-05 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944872+0000", 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9445e-05 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147126+0000", 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202254668 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147523+0000", 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039669999999999999 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.as0 65/63 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:24:51.838 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.858262042, 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20196552700000001, 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945736+0000", 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945793+0000", 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7462e-05 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945835+0000", 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1705999999999997e-05 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945854+0000", 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9298999999999999e-05 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147627+0000", 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20177293299999999 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147699+0000", 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2384000000000004e-05 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 66/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.857419082, 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201296893, 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.842 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946579+0000", 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0360000000000001e-06 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946590+0000", 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1583000000000001e-05 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946624+0000", 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3849000000000001e-05 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946648+0000", 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3326999999999999e-05 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147789+0000", 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201141864 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147874+0000", 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4233999999999993e-05 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.as0 66/63 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.857255806, 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30137683399999998, 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.846 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.847 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946741+0000", 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7599999999999997e-07 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946762+0000", 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0993e-05 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946792+0000", 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9776000000000001e-05 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946814+0000", 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2889999999999999e-05 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.248046+0000", 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30123198000000001 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.248117+0000", 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0518999999999993e-05 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.849 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 66/63 ECSubWriteReply(tid=26, last_complete=66'21, committed=1, applied=1))", 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.455416+0000", 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 13.348579855000001, 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200451711, 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455416+0000", 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455416+0000", 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455418+0000", 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.697e-06 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455425+0000", 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5830000000000001e-06 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.455454+0000", 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8889000000000001e-05 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.555551+0000", 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10009688 2024-04-26T01:24:51.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.655817+0000", 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.1002654 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.655868+0000", 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1261999999999998e-05 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.853 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/63 ECSubWriteReply(tid=38, last_complete=77'33, committed=1, applied=1))", 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.213626+0000", 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.590370348, 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26814398900000003, 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213626+0000", 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213626+0000", 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213627+0000", 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8759999999999999e-06 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213637+0000", 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7799999999999995e-06 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213682+0000", 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4736e-05 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.313774+0000", 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100091633 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.481747+0000", 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16797363800000001 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.481770+0000", 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2325999999999999e-05 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/63 ECSubWriteReply(tid=37, last_complete=77'32, committed=1, applied=1))", 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.403943+0000", 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.40005281, 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20049687799999999, 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.857 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403943+0000", 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403943+0000", 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403943+0000", 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0699999999999998e-07 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403946+0000", 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9699999999999999e-06 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403955+0000", 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7730000000000005e-06 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403962+0000", 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8059999999999997e-06 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.604405+0000", 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044329899999999 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.604440+0000", 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4722999999999999e-05 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/63 ECSubWriteReply(tid=38, last_complete=77'33, committed=1, applied=1))", 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.403975+0000", 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.4000204909999998, 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200535188, 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403975+0000", 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403975+0000", 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403976+0000", 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1e-07 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403979+0000", 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9270000000000001e-06 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.403990+0000", 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1389999999999999e-05 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.504141+0000", 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100151219 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.604486+0000", 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10034457300000001 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.604511+0000", 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4769e-05 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.864 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/63 ECSubWriteReply(tid=39, last_complete=77'33, committed=1, applied=1))", 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.807488+0000", 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.996508103, 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20182555899999999, 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807488+0000", 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807488+0000", 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807489+0000", 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3629999999999999e-06 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807496+0000", 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7859999999999999e-06 2024-04-26T01:24:51.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807515+0000", 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8839999999999999e-05 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807525+0000", 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0693000000000001e-05 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.009283+0000", 2024-04-26T01:24:51.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20175726799999999 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.009313+0000", 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0608999999999998e-05 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 78/30 ECSubWrite(tid=222, reqid=client.4589.0:680, at_version=78'57, trim_to=0'0, roll_forward_to=72'53))", 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.064452+0000", 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.739543847, 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29624553599999998, 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.868 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064452+0000", 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064452+0000", 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064454+0000", 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9130000000000001e-06 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064618+0000", 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016357700000000001 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064648+0000", 2024-04-26T01:24:51.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0788999999999999e-05 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064673+0000", 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4620000000000001e-05 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.258550+0000", 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.193877522 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.258567+0000", 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6274e-05 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360645+0000", 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10207862199999999 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360698+0000", 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2219000000000003e-05 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.872 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 78/30 ECSubWrite(tid=223, reqid=client.4589.0:681, at_version=78'58, trim_to=0'0, roll_forward_to=72'53))", 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.064720+0000", 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.7392757859999999, 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29601289400000003, 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064720+0000", 2024-04-26T01:24:51.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064720+0000", 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064722+0000", 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6700000000000001e-06 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064889+0000", 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000166773 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064910+0000", 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1373e-05 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064929+0000", 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9179999999999999e-05 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.259402+0000", 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19447310500000001 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.259416+0000", 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3956e-05 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360704+0000", 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101287821 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360733+0000", 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9016000000000001e-05 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 78/30 ECSubWrite(tid=224, reqid=client.4589.0:682, at_version=78'59, trim_to=0'0, roll_forward_to=72'53))", 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.064977+0000", 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.7390185740000002, 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29639301600000001, 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064977+0000", 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064977+0000", 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064979+0000", 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5400000000000001e-06 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.065003+0000", 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4332e-05 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.065020+0000", 2024-04-26T01:24:51.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6507e-05 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.065038+0000", 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7796e-05 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360313+0000", 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.295275909 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360330+0000", 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6192e-05 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.361295+0000", 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00096553199999999996 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.361370+0000", 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5208000000000002e-05 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 78/63 ECSubWrite(tid=38, reqid=client.4589.0:693, at_version=78'62, trim_to=0'0, roll_forward_to=72'60))", 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.234436+0000", 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.569559489, 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20538144699999999, 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234436+0000", 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234436+0000", 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234438+0000", 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1400000000000001e-06 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234476+0000", 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7993e-05 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234496+0000", 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0543e-05 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.334633+0000", 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100136776 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.435049+0000", 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100416124 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.435066+0000", 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6816999999999998e-05 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.439736+0000", 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046703090000000001 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.439818+0000", 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1744999999999997e-05 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.885 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.886 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:24:51.886 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:24:51.886 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:51.887 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:51.887 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:51.888 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:51.888 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:51.889 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:51.889 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:51.889 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:51.890 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:51.890 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:51.890 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:51.891+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:51.895 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:51.896 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:51.915 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:24:51.916 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T01:24:51.916 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:51.917 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:51.921 INFO:tasks.rados.rados.0.smithi181.stdout:274: done (5 left) 2024-04-26T01:24:51.921 INFO:tasks.rados.rados.0.smithi181.stdout:275: done (4 left) 2024-04-26T01:24:51.921 INFO:tasks.rados.rados.0.smithi181.stdout:276: done (3 left) 2024-04-26T01:24:51.921 INFO:tasks.rados.rados.0.smithi181.stdout:277: done (2 left) 2024-04-26T01:24:51.921 INFO:tasks.rados.rados.0.smithi181.stdout:278: done (1 left) 2024-04-26T01:24:51.922 INFO:tasks.rados.rados.0.smithi181.stdout:280: done (0 left) 2024-04-26T01:24:51.922 INFO:tasks.rados.rados.0.smithi181.stdout:281: snap_create 2024-04-26T01:24:51.940 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:51.940 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:51.941 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-26T01:24:51.960 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:51.960 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:51.978 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:51.978 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:51.988 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:51.991+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:51.996 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:51.996 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:52.016 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:52.016 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:52.017 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:52.017 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:52.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:52.038 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:52.038 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:52.056 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:52.057 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:52.088 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:24:52.091+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:24:52.189 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:52.191+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:52.194 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:52.194 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:52.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:24:52.291 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:52.295+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:52.331 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:52.331 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:52.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:24:52.392 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:52.395+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:52.471 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:52.471 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:52.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:24:52.532 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:52.537+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:52.593 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:52.595+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:52.601 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:52.602 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:52.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:24:52.694 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:52.699+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:52.784 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:52.784 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:52.795 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:52.797+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:52.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:24:52.895 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:52.901+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:52.934 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.8 to 1 (10000) 2024-04-26T01:24:52.941 INFO:tasks.rados.rados.0.smithi181.stdout:281: done (0 left) 2024-04-26T01:24:52.941 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 30 current snap is 23 2024-04-26T01:24:52.942 INFO:tasks.rados.rados.0.smithi181.stdout:282: seq_num 109 ranges {4145152=671744,4816896=753664,5570560=409600} 2024-04-26T01:24:52.961 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-26T01:24:52.962 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:52.962 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:52.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:24:52.996 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:52.999+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:53.002 INFO:tasks.rados.rados.0.smithi181.stdout:282: writing smithi18128752-30 from 4145152 to 4816896 tid 1 2024-04-26T01:24:53.010 INFO:tasks.rados.rados.0.smithi181.stdout:282: writing smithi18128752-30 from 4816896 to 5570560 tid 2 2024-04-26T01:24:53.014 INFO:tasks.rados.rados.0.smithi181.stdout:282: writing smithi18128752-30 from 5570560 to 5980160 tid 3 2024-04-26T01:24:53.014 INFO:tasks.rados.rados.0.smithi181.stdout:283: rmattr oid 17 current snap is 23 2024-04-26T01:24:53.014 INFO:tasks.rados.rados.0.smithi181.stdout:283: done (1 left) 2024-04-26T01:24:53.014 INFO:tasks.rados.rados.0.smithi181.stdout:284: snap_remove snap 17 2024-04-26T01:24:53.097 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:53.101+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:53.127 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:53.127 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:53.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:24:53.199 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:53.199+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:53.300 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:53.303+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:53.304 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:53.304 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:53.318 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:24:53.401 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:53.403+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:53.440 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:53.440 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:53.454 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:24:53.502 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:53.507+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:53.580 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:53.580 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:53.594 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:24:53.639 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:53.643+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:53.704 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:53.707+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:53.720 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:53.720 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:53.734 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:24:53.776 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.9 to 1 (10000) 2024-04-26T01:24:53.782 INFO:tasks.rados.rados.0.smithi181.stdout:284: done (1 left) 2024-04-26T01:24:53.782 INFO:tasks.rados.rados.0.smithi181.stdout:285: snap_remove snap 21 2024-04-26T01:24:53.798 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-26T01:24:53.805 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:53.807+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:53.906 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:53.911+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:53.914 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:53.914 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:53.929 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:24:53.943 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:24:54.007 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:54.011+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:54.076 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:54.093 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:54.093 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:54.108 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:54.111+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:54.114 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:24:54.209 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:54.211+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:54.277 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:54.277 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:54.291 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:24:54.306 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:54.306 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:54.310 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:54.311+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:54.328 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:54.328 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:54.350 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:54.350 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:54.378 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:54.379 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:54.401 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:54.401 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:54.408 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:54.408 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:54.411 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:54.417+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:54.422 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:24:54.512 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:54.513 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:54.513 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:54.515+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:54.549 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:54.549 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:54.585 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:54.586 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:54.614 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:54.617+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:54.616 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:54.616 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:54.627 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:54.627 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:54.627 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:54.627 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:54.627 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:54.645 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:24:54.649 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:54.650 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:54.670 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:54.670 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:54.690 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:54.690 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:54.706 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:54.706 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:54.714 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:54.715+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:54.725 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:54.726 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:54.745 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:54.746 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:54.766 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:54.766 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:185 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.704667653999998, 2024-04-26T01:24:54.776 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.395059522, 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 185 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.777 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073211+0000", 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073214+0000", 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.778 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073225+0000", 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0404999999999999e-05 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.073253+0000", 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8058000000000001e-05 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173330+0000", 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10007712000000001 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.779 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.261568+0000", 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.088237503999999994 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.261653+0000", 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5259999999999993e-05 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.262369+0000", 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00071622000000000005 2024-04-26T01:24:54.780 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265729+0000", 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.003360264 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.468152+0000", 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20242310199999999 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.468271+0000", 2024-04-26T01:24:54.781 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000118279 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(osd.4.21:2 3.4s0 3:2e10a639:::smithi18128752-28:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.617790214000003, 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30734204100000001, 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:54.782 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160089+0000", 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160090+0000", 2024-04-26T01:24:54.783 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7129999999999999e-06 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160104+0000", 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3596e-05 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160114+0000", 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8190000000000003e-06 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.784 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160125+0000", 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1413999999999999e-05 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160185+0000", 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9914000000000003e-05 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.160251+0000", 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5728999999999994e-05 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467431+0000", 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30717985599999997 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:186 3.4s0 3:2be1e129:::smithi18128752-30:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.604531539, 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39528098299999997, 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.786 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 186 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.787 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173347+0000", 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173348+0000", 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.108e-06 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173354+0000", 2024-04-26T01:24:54.788 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3009999999999997e-06 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173362+0000", 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9710000000000004e-06 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173367+0000", 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3179999999999997e-06 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.789 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263197+0000", 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.089829862999999996 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263286+0000", 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9519999999999997e-05 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.263786+0000", 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00050013300000000004 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.790 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265873+0000", 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.002086606 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568417+0000", 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30254369399999997 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.568628+0000", 2024-04-26T01:24:54.791 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021146899999999999 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:187 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.604504321, 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51780501899999998, 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.792 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 187 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:24:54.793 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173375+0000", 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1699999999999999e-07 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173377+0000", 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.939e-06 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.794 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173381+0000", 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2050000000000004e-06 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173384+0000", 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2760000000000001e-06 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264258+0000", 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.090873535000000005 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.795 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264532+0000", 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027448400000000002 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265351+0000", 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081914900000000003 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.466871+0000", 2024-04-26T01:24:54.796 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20151915000000001 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690961+0000", 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22409068500000001 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.691180+0000", 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021827899999999999 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.797 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 37/30 ECSubWriteReply(tid=14, last_complete=37'9, committed=1, applied=1))", 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.513198627999998, 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.203106276, 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.798 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264680+0000", 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264681+0000", 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5999999999999991e-07 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.799 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264687+0000", 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9549999999999997e-06 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264702+0000", 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.486e-05 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.364796+0000", 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100093734 2024-04-26T01:24:54.800 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467752+0000", 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102956666 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.467786+0000", 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4100999999999998e-05 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.801 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:270 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 42.604965329000002, 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22773885299999999, 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 270 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.802 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172913+0000", 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172916+0000", 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0170000000000001e-06 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.803 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172932+0000", 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5766000000000001e-05 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172963+0000", 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0440999999999999e-05 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.172980+0000", 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7224e-05 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.804 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.295355+0000", 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.122375555 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.295455+0000", 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9192999999999997e-05 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.400652+0000", 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.105197657 2024-04-26T01:24:54.805 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:271 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 42.604862857999997, 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53167738200000003, 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.806 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 271 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.807 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173017+0000", 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1100000000000005e-07 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173022+0000", 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4990000000000002e-06 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173034+0000", 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1667e-05 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.808 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173044+0000", 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9070000000000002e-06 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398177+0000", 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225132687 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398388+0000", 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000211325 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.809 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.704693+0000", 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30630548600000002 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:272 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 42.604821178999998, 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53273070499999997, 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 272 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9200000000000001e-07 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173079+0000", 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1137000000000001e-05 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173093+0000", 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3198e-05 2024-04-26T01:24:54.812 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173102+0000", 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1859999999999998e-06 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401582+0000", 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22848043000000001 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401813+0000", 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023127900000000001 2024-04-26T01:24:54.813 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.705788+0000", 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30397488299999997 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s4 61/30 ECSubWrite(tid=177, reqid=client.4589.0:478, at_version=61'44, trim_to=0'0, roll_forward_to=59'40))", 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.899783+0000", 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 21.878095417000001, 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20341484900000001, 2024-04-26T01:24:54.814 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899783+0000", 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899783+0000", 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.815 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899785+0000", 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3230000000000001e-06 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899883+0000", 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7841000000000001e-05 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.899904+0000", 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.105e-05 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.816 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.999981+0000", 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10007708899999999 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.100182+0000", 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100201789 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.100210+0000", 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7166e-05 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.817 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.103164+0000", 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0029540679999999998 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.103198+0000", 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4523000000000001e-05 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.818 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es4 67/44 ECSubWrite(tid=257, reqid=osd.13.0:257, at_version=67'47, trim_to=0'0, roll_forward_to=61'45))", 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.896187+0000", 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.881691967, 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20555757099999999, 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.896187+0000", 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.819 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.896187+0000", 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.896187+0000", 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.49e-07 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.896211+0000", 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3574000000000002e-05 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.820 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.822 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.822 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.896223+0000", 2024-04-26T01:24:54.822 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2911000000000001e-05 2024-04-26T01:24:54.822 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.822 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.822 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.822 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.996294+0000", 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10007042300000001 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.000422+0000", 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.004127889 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.000428+0000", 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6980000000000001e-06 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.101698+0000", 2024-04-26T01:24:54.823 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101269463 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.101744+0000", 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6464000000000001e-05 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:545 3.4s0 3:2be1e129:::smithi18128752-30:head [stat,delete] snapc 12=[12,10,e] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:39.784981+0000", 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.992897591, 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30808959299999999, 2024-04-26T01:24:54.824 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 545 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.784981+0000", 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.825 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.784981+0000", 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.784985+0000", 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5389999999999999e-06 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.784999+0000", 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4216000000000001e-05 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.826 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.785029+0000", 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9686e-05 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.785043+0000", 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4052e-05 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.785200+0000", 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015729700000000001 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.827 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.785217+0000", 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6730000000000001e-05 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.888681+0000", 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10346408 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.888860+0000", 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000179526 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.828 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.890322+0000", 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001461501 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.891793+0000", 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014710909999999999 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:40.093007+0000", 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20121410100000001 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.829 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:40.093071+0000", 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3774000000000003e-05 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:611 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~540672] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.812212+0000", 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.965667262, 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50597818100000003, 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.830 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 611 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.812212+0000", 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.812212+0000", 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.831 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.812214+0000", 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6769999999999999e-06 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.813057+0000", 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00084302800000000001 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.813088+0000", 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0865999999999999e-05 2024-04-26T01:24:54.832 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.813102+0000", 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4239999999999999e-05 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913298+0000", 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100195669 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913306+0000", 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6909999999999997e-06 2024-04-26T01:24:54.833 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.914357+0000", 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00105167 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.914510+0000", 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015237000000000001 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.916163+0000", 2024-04-26T01:24:54.834 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001653259 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921172+0000", 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050092299999999999 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.317916+0000", 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.396743868 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.318190+0000", 2024-04-26T01:24:54.835 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027361300000000002 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:612 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~507904] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.816201+0000", 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.961677723999999, 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50253841700000002, 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.836 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 612 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816201+0000", 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816201+0000", 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.837 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816206+0000", 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3460000000000001e-06 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816745+0000", 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00053856399999999995 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816773+0000", 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8017e-05 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.838 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816787+0000", 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3854e-05 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913509+0000", 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096722404999999997 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913519+0000", 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6800000000000005e-06 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.839 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917532+0000", 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040130610000000001 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917712+0000", 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000180156 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.919317+0000", 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001604891 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.840 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921251+0000", 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00193424 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.318533+0000", 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39728167399999997 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.318739+0000", 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000206529 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.841 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:613 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~622592] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.821128+0000", 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.956750756, 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59873126099999996, 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 613 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.842 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.821128+0000", 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.821128+0000", 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.821131+0000", 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5239999999999999e-06 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.843 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822321+0000", 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001190485 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822353+0000", 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2147000000000002e-05 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822369+0000", 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5593000000000001e-05 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.844 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921300+0000", 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098931370000000005 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921469+0000", 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016842499999999999 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.922991+0000", 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001522036 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.845 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.927109+0000", 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0041182099999999998 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.419729+0000", 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.49262054799999999 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.419859+0000", 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000129923 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.846 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:614 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~32768] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822397+0000", 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.955481404, 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59784716400000004, 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 614 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.847 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822397+0000", 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822397+0000", 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822399+0000", 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2529999999999999e-06 2024-04-26T01:24:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822459+0000", 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0736000000000003e-05 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822478+0000", 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8193999999999999e-05 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822488+0000", 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.028e-05 2024-04-26T01:24:54.849 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.924397+0000", 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101909182 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.924681+0000", 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028437299999999998 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.926252+0000", 2024-04-26T01:24:54.850 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015702299999999999 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929400+0000", 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031478809999999999 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420107+0000", 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.490707952 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420245+0000", 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000137083 2024-04-26T01:24:54.851 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:615 3.4s0 3:2e10a639:::smithi18128752-28:head [setxattr _header (57) in=7b] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822505+0000", 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.955373965, 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59808836700000001, 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.852 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 615 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822505+0000", 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822505+0000", 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.853 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822506+0000", 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8000000000000004e-07 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822511+0000", 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8369999999999996e-06 2024-04-26T01:24:54.854 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822524+0000", 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.312e-05 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822533+0000", 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.02e-06 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.855 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.927335+0000", 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.104802701 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.927618+0000", 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028260900000000002 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.928984+0000", 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001365851 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.856 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930534+0000", 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001550608 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420432+0000", 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.48989737799999999 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420593+0000", 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016136300000000001 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.857 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:616 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822544+0000", 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.955334701, 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80089191999999998, 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.858 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 616 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822544+0000", 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822544+0000", 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.859 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822545+0000", 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5999999999999991e-07 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822550+0000", 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2299999999999999e-06 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822564+0000", 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3499999999999999e-05 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.860 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822572+0000", 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4200000000000007e-06 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929502+0000", 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.106929992 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929766+0000", 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026414899999999998 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.861 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420729+0000", 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.49096291800000003 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420898+0000", 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016868300000000001 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.623436+0000", 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20253806799999999 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.862 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:617 3.4s0 3:2be1e129:::smithi18128752-30:head [copy-from ver 59] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822584+0000", 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.955294524999999, 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0342352619999999, 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 617 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.863 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822584+0000", 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822584+0000", 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822585+0000", 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3e-07 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.864 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822590+0000", 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.707e-06 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822602+0000", 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.243e-05 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822610+0000", 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7030000000000006e-06 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.865 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929881+0000", 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.107270828 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930159+0000", 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027794499999999998 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.647173+0000", 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71701393899999999 2024-04-26T01:24:54.866 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653979+0000", 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0068063760000000003 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.856678+0000", 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20269896000000001 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.856820+0000", 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014164399999999999 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.867 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:618 3.4s0 3:2be1e129:::smithi18128752-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.955258085000001, 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.83091384000000001, 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 618 2024-04-26T01:24:54.868 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:24:54.869 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6700000000000003e-07 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822625+0000", 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3900000000000002e-06 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822637+0000", 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2033e-05 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822645+0000", 2024-04-26T01:24:54.870 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6559999999999992e-06 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930630+0000", 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.107985175 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930789+0000", 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015969000000000001 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653189+0000", 2024-04-26T01:24:54.871 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.72239973400000002 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653353+0000", 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016386600000000001 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653535+0000", 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018152899999999999 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds5 71/63 ECSubWrite(tid=26, reqid=client.4589.0:598, at_version=0'0, trim_to=0'0, roll_forward_to=71'56))", 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.917388+0000", 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 10.860490823999999, 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20230589299999999, 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917388+0000", 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.873 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917388+0000", 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917389+0000", 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8899999999999999e-07 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917410+0000", 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1784e-05 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.874 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917442+0000", 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1396000000000002e-05 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917465+0000", 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3223000000000001e-05 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.018956+0000", 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101491308 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.875 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.018993+0000", 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6563000000000002e-05 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.119634+0000", 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100640753 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.119694+0000", 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0177000000000003e-05 2024-04-26T01:24:54.876 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.877 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.877 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.877 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:54.877 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:54.877 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:54.879 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.10 to 1 (10000) 2024-04-26T01:24:54.880 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:285: done (1 left) 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 6 current snap is 23 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:286: seq_num 110 ranges {98304=688128,786432=507904,1294336=655360} 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:286: writing smithi18128752-6 from 98304 to 786432 tid 1 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:286: writing smithi18128752-6 from 786432 to 1294336 tid 2 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:286: writing smithi18128752-6 from 1294336 to 1949696 tid 3 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:287: read oid 11 snap -1 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:287: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 28 current snap is 23 2024-04-26T01:24:54.881 INFO:tasks.rados.rados.0.smithi181.stdout:288: seq_num 111 ranges {2162688=704512,2867200=655360,3522560=507904} 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:288: writing smithi18128752-28 from 2162688 to 2867200 tid 1 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:288: writing smithi18128752-28 from 2867200 to 3522560 tid 2 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:288: writing smithi18128752-28 from 3522560 to 4030464 tid 3 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:289: snap_create 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 1 to smithi18128752-30 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 2 to smithi18128752-30 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 3 to smithi18128752-30 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 4 to smithi18128752-30 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 5 to smithi18128752-30 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 30 updating version 0 to 63 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 30 version 63 is already newer than 60 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 30 version 63 is already newer than 61 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 30 version 63 is already newer than 62 2024-04-26T01:24:54.882 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 63 (ObjNum 109 snap 23 seq_num 109) dirty exists 2024-04-26T01:24:54.883 INFO:tasks.rados.rados.0.smithi181.stdout:282: left oid 30 (ObjNum 109 snap 23 seq_num 109) 2024-04-26T01:24:54.883 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:24:54.884 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-26T01:24:54.885 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:24:54.883+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:24:54.979 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:54.983+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:55.049 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:55.050 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:55.050 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:24:55.050 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:55.065 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:24:55.079 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:55.083+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:55.175 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:55.175 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:24:55.175 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:24:55.175 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:24:55.175 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:55.181 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:55.185+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:55.188 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:24:55.207 INFO:tasks.rados.rados.0.smithi181.stdout:286: finishing write tid 1 to smithi18128752-6 2024-04-26T01:24:55.258 INFO:tasks.rados.rados.0.smithi181.stdout:288: finishing write tid 1 to smithi18128752-28 2024-04-26T01:24:55.259 INFO:tasks.rados.rados.0.smithi181.stdout:288: finishing write tid 2 to smithi18128752-28 2024-04-26T01:24:55.259 INFO:tasks.rados.rados.0.smithi181.stdout:288: finishing write tid 3 to smithi18128752-28 2024-04-26T01:24:55.259 INFO:tasks.rados.rados.0.smithi181.stdout:288: finishing write tid 4 to smithi18128752-28 2024-04-26T01:24:55.283 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:55.287+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:55.307 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.6s4 35/34 ECSubRead(tid=8, to_read={3:60f0095a:::smithi18128752-3:head=0,262144,0}, subchunks={3:60f0095a:::smithi18128752-3:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 48.893095250000002, 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100734536, 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.308 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420052+0000", 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.846e-06 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.309 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420069+0000", 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7493e-05 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420098+0000", 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9079000000000001e-05 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420118+0000", 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9468999999999999e-05 2024-04-26T01:24:55.310 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520369+0000", 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100251189 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520785+0000", 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00041545999999999998 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.311 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.es6 36/34 ECSubRead(tid=24, to_read={3:738de3f3:::smithi18128752-9:head=0,262144,0}, subchunks={3:738de3f3:::smithi18128752-9:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 47.593403549999998, 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00074145500000000002, 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.312 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719742+0000", 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.03e-07 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719756+0000", 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4053000000000001e-05 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719789+0000", 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2913999999999997e-05 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719810+0000", 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0513000000000002e-05 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.719965+0000", 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015534199999999999 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:24:55.314 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.720483+0000", 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051822999999999995 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:24:55.315 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:24:55.322 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:24:55.384 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:55.389+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:55.407 INFO:tasks.rados.rados.0.smithi181.stdout:286: finishing write tid 2 to smithi18128752-6 2024-04-26T01:24:55.408 INFO:tasks.rados.rados.0.smithi181.stdout:286: finishing write tid 3 to smithi18128752-6 2024-04-26T01:24:55.409 INFO:tasks.rados.rados.0.smithi181.stdout:286: finishing write tid 4 to smithi18128752-6 2024-04-26T01:24:55.416 INFO:tasks.rados.rados.0.smithi181.stdout:287: expect (ObjNum 10 snap 0 seq_num 10) 2024-04-26T01:24:55.457 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:55.457 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:55.457 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-04-26T01:24:55.457 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:55.460 INFO:tasks.rados.rados.0.smithi181.stdout:288: finishing write tid 5 to smithi18128752-28 2024-04-26T01:24:55.460 INFO:tasks.rados.rados.0.smithi181.stdout:288: oid 28 updating version 0 to 64 2024-04-26T01:24:55.460 INFO:tasks.rados.rados.0.smithi181.stdout:288: oid 28 updating version 64 to 66 2024-04-26T01:24:55.460 INFO:tasks.rados.rados.0.smithi181.stdout:288: oid 28 updating version 66 to 67 2024-04-26T01:24:55.460 INFO:tasks.rados.rados.0.smithi181.stdout:288: oid 28 version 67 is already newer than 65 2024-04-26T01:24:55.460 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 28 v 67 (ObjNum 111 snap 23 seq_num 111) dirty exists 2024-04-26T01:24:55.460 INFO:tasks.rados.rados.0.smithi181.stdout:288: left oid 28 (ObjNum 111 snap 23 seq_num 111) 2024-04-26T01:24:55.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:24:55.518 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:55.523+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:55.585 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:55.587+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:55.598 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:55.598 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:55.598 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:55.612 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:24:55.613 INFO:tasks.rados.rados.0.smithi181.stdout:286: finishing write tid 5 to smithi18128752-6 2024-04-26T01:24:55.613 INFO:tasks.rados.rados.0.smithi181.stdout:286: oid 6 updating version 0 to 34 2024-04-26T01:24:55.613 INFO:tasks.rados.rados.0.smithi181.stdout:286: oid 6 updating version 34 to 35 2024-04-26T01:24:55.613 INFO:tasks.rados.rados.0.smithi181.stdout:286: oid 6 version 35 is already newer than 32 2024-04-26T01:24:55.613 INFO:tasks.rados.rados.0.smithi181.stdout:286: oid 6 version 35 is already newer than 33 2024-04-26T01:24:55.614 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 35 (ObjNum 110 snap 23 seq_num 110) dirty exists 2024-04-26T01:24:55.614 INFO:tasks.rados.rados.0.smithi181.stdout:286: left oid 6 (ObjNum 110 snap 23 seq_num 110) 2024-04-26T01:24:55.685 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:55.687+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=11, reqid=client.4589.0:127, at_version=35'7, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.208367123999999, 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202990695, 2024-04-26T01:24:55.784 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.785 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577859+0000", 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.013e-06 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578257+0000", 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039834700000000001 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578289+0000", 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2016e-05 2024-04-26T01:24:55.786 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578306+0000", 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6460000000000002e-05 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678641+0000", 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100335199 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678656+0000", 2024-04-26T01:24:55.787 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5233e-05 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780779+0000", 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102122955 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780847+0000", 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8472000000000004e-05 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.788 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=31, reqid=client.4589.0:124, at_version=35'13, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.934845871, 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20482444299999999, 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.789 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851380+0000", 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.973e-06 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852052+0000", 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067198400000000004 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.790 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852084+0000", 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2298999999999997e-05 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952200+0000", 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100116018 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952352+0000", 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015207299999999999 2024-04-26T01:24:55.791 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952377+0000", 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4363e-05 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056116+0000", 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103738862 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056202+0000", 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6871000000000002e-05 2024-04-26T01:24:55.792 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 35/30 ECSubWrite(tid=170, reqid=client.4589.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.816909383000002, 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20178349000000001, 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.793 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969315+0000", 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969317+0000", 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0600000000000002e-06 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.794 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969365+0000", 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8296e-05 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969439+0000", 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3907999999999998e-05 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.969466+0000", 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7223e-05 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.795 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169873+0000", 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20040670999999999 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.169887+0000", 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4727e-05 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171018+0000", 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001130968 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.796 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.171098+0000", 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9598000000000006e-05 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.509138597000003, 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20597036399999999, 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.797 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.798 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277086+0000", 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277250+0000", 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016446899999999999 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277265+0000", 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.418e-05 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.799 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277275+0000", 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0456e-05 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481015+0000", 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203739686 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481042+0000", 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6962999999999999e-05 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.800 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482955+0000", 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001913816 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483056+0000", 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100204 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.801 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.4s3 41/30 ECSubWrite(tid=22, reqid=osd.8.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=41'12))", 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 43.389671921000001, 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20184675299999999, 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.802 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396552+0000", 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396553+0000", 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.31e-06 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396576+0000", 2024-04-26T01:24:55.803 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2816e-05 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396597+0000", 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1166999999999999e-05 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.806 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.396611+0000", 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4099e-05 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.496984+0000", 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100372218 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.497002+0000", 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8051999999999999e-05 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.807 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.598348+0000", 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10134613200000001 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.598399+0000", 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0958999999999998e-05 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/44 ECSubWrite(tid=71, reqid=client.4589.0:315, at_version=0'0, trim_to=0'0, roll_forward_to=46'20))", 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:55.808 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 37.530789657, 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30303612499999999, 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:24:55.809 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255436+0000", 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.81e-06 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255469+0000", 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2895999999999998e-05 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255499+0000", 2024-04-26T01:24:55.810 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0253e-05 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.355601+0000", 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100101499 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455871+0000", 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10027049 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.811 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455899+0000", 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7636000000000001e-05 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558401+0000", 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102502418 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558470+0000", 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9122999999999995e-05 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.812 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=68, reqid=client.4589.0:383, at_version=51'42, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.467675354000001, 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.302538633, 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.813 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318551+0000", 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0930000000000001e-06 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318918+0000", 2024-04-26T01:24:55.814 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036699299999999997 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318977+0000", 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9203999999999998e-05 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419092+0000", 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100115619 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419200+0000", 2024-04-26T01:24:55.815 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010721300000000001 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419220+0000", 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0363000000000001e-05 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620972+0000", 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201752242 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621087+0000", 2024-04-26T01:24:55.816 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114906 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=206, reqid=client.4589.0:388, at_version=51'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.444143015000002, 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.300923198, 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.817 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342083+0000", 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.793e-06 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.818 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342388+0000", 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030580700000000001 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342408+0000", 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9683000000000001e-05 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342417+0000", 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2550000000000003e-06 2024-04-26T01:24:55.819 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.442698+0000", 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100280522 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.442716+0000", 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7808999999999998e-05 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.642922+0000", 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20020643499999999 2024-04-26T01:24:55.820 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.643004+0000", 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1893999999999999e-05 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=69, reqid=client.4589.0:384, at_version=51'43, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.367087941000001, 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20207976799999999, 2024-04-26T01:24:55.821 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.822 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419137+0000", 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.54e-07 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419193+0000", 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6375999999999997e-05 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419206+0000", 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.323e-05 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.823 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419217+0000", 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0906e-05 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519979+0000", 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10076136199999999 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519989+0000", 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0507e-05 2024-04-26T01:24:55.824 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621131+0000", 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10114197 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621216+0000", 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4662999999999999e-05 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.825 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=70, reqid=client.4589.0:385, at_version=51'44, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.366987731000002, 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20264654900000001, 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.826 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419237+0000", 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8400000000000005e-07 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419295+0000", 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8211999999999998e-05 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.827 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419307+0000", 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2558000000000001e-05 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519470+0000", 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100162563 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620565+0000", 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10109543 2024-04-26T01:24:55.828 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620584+0000", 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8352e-05 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621809+0000", 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012251160000000001 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621883+0000", 2024-04-26T01:24:55.829 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3833999999999998e-05 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=207, reqid=client.4589.0:389, at_version=51'19, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.344804924999998, 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202618775, 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.830 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441420+0000", 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1489999999999999e-06 2024-04-26T01:24:55.831 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441473+0000", 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2834999999999998e-05 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441492+0000", 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9145000000000001e-05 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441503+0000", 2024-04-26T01:24:55.832 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1151e-05 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.641827+0000", 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200323321 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.641854+0000", 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7695000000000002e-05 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.833 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.643949+0000", 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020951979999999999 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.644038+0000", 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8281000000000004e-05 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=209, reqid=client.4589.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=51'20))", 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:24:55.834 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 31.937961588, 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.378972751, 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:24:55.835 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848264+0000", 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3510000000000001e-06 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848286+0000", 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2011999999999999e-05 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.836 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848307+0000", 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1798999999999999e-05 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848322+0000", 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4968e-05 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848406+0000", 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3762000000000002e-05 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.837 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848418+0000", 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1626e-05 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.227182+0000", 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.37876395099999999 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.227235+0000", 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3282000000000002e-05 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.838 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 56/44 ECSubWrite(tid=107, reqid=osd.14.0:107, at_version=56'36, trim_to=0'0, roll_forward_to=55'33))", 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:28.148898+0000", 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 27.637325881999999, 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20199924699999999, 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.839 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148898+0000", 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148898+0000", 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148899+0000", 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1699999999999998e-07 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.840 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148934+0000", 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5234999999999997e-05 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.148989+0000", 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5135e-05 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.249123+0000", 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100133705 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.349389+0000", 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.1002661 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.349404+0000", 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5345999999999998e-05 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.350807+0000", 2024-04-26T01:24:55.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014034119999999999 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:28.350897+0000", 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9796999999999994e-05 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 61/44 ECSubWrite(tid=127, reqid=client.4589.0:471, at_version=61'56, trim_to=0'0, roll_forward_to=59'47))", 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.971787+0000", 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.814436573999998, 2024-04-26T01:24:55.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20188065899999999, 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971787+0000", 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971787+0000", 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971789+0000", 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.606e-06 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971822+0000", 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2762999999999998e-05 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.971849+0000", 2024-04-26T01:24:55.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7503000000000001e-05 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.071951+0000", 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10010151 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.172207+0000", 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10025627500000001 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.846 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.172233+0000", 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6004999999999999e-05 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.173576+0000", 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001342789 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.173668+0000", 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2207999999999995e-05 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.847 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/44 ECSubWrite(tid=146, reqid=osd.14.0:146, at_version=67'60, trim_to=0'0, roll_forward_to=61'58))", 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.995901572000001, 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20292310799999999, 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:24:55.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790323+0000", 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1299999999999999e-07 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790372+0000", 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8862000000000001e-05 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790402+0000", 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9595999999999999e-05 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790435+0000", 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3872999999999999e-05 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.891521+0000", 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10108587600000001 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.891538+0000", 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6816999999999998e-05 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.993158+0000", 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101619426 2024-04-26T01:24:55.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.993245+0000", 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7844999999999997e-05 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s2 67/44 ECSubRead(tid=111, to_read={3:1fe6984d:::smithi18128752-48:head=0,811008,0}, subchunks={3:1fe6984d:::smithi18128752-48:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.988927884999999, 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.296535041, 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.852 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.853 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797298+0000", 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.66e-06 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797315+0000", 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7555999999999999e-05 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797346+0000", 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1192999999999998e-05 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797375+0000", 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8484999999999999e-05 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.093325+0000", 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29594971799999997 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.093831+0000", 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050642899999999997 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.855 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/44 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=67'64, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.451206042999999, 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20216706400000001, 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3700000000000001e-07 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335038+0000", 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0047e-05 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335047+0000", 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7329999999999992e-06 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.435145+0000", 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100097939 2024-04-26T01:24:55.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.535402+0000", 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10025746200000001 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.535419+0000", 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6529e-05 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.537144+0000", 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017251930000000001 2024-04-26T01:24:55.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.537185+0000", 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0824000000000001e-05 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s2 72/63 ECSubWrite(tid=54, reqid=osd.0.0:54, at_version=72'37, trim_to=0'0, roll_forward_to=71'35))", 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.874441073, 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20240396699999999, 2024-04-26T01:24:55.860 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911840+0000", 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6796000000000003e-05 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911867+0000", 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7155999999999999e-05 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911891+0000", 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3195999999999999e-05 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.112540+0000", 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200648943 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.112562+0000", 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2189999999999999e-05 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.114107+0000", 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015455899999999999 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.114187+0000", 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9425999999999995e-05 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 72/44 ECSubWrite(tid=131, reqid=osd.5.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=72'94))", 2024-04-26T01:24:55.864 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:44.779998+0000", 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 11.006226241, 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20524366399999999, 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.779998+0000", 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.779998+0000", 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.780000+0000", 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.883e-06 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.780028+0000", 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8333000000000001e-05 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.780066+0000", 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7942999999999999e-05 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.880189+0000", 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100123209 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.982504+0000", 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102314949 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.982519+0000", 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.985126+0000", 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026065659999999998 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.985241+0000", 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011561399999999999 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 78/44 ECSubWrite(tid=295, reqid=client.4589.0:685, at_version=78'63, trim_to=0'0, roll_forward_to=72'61))", 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:50.188947+0000", 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.5972771980000005, 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202856176, 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.188947+0000", 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.188947+0000", 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.188948+0000", 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7540000000000001e-06 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.189225+0000", 2024-04-26T01:24:55.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000276758 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.189258+0000", 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3195999999999999e-05 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.189292+0000", 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3692999999999997e-05 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.289603+0000", 2024-04-26T01:24:55.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100310776 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.289619+0000", 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6013000000000001e-05 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.391740+0000", 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10212062399999999 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.391803+0000", 2024-04-26T01:24:55.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3361999999999994e-05 2024-04-26T01:24:55.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.873 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:55.873 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:55.873 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:55.874 INFO:tasks.rados.rados.0.smithi181.stdout:282: done (4 left) 2024-04-26T01:24:55.875 INFO:tasks.rados.rados.0.smithi181.stdout:286: done (3 left) 2024-04-26T01:24:55.875 INFO:tasks.rados.rados.0.smithi181.stdout:287: done (2 left) 2024-04-26T01:24:55.875 INFO:tasks.rados.rados.0.smithi181.stdout:288: done (1 left) 2024-04-26T01:24:55.875 INFO:tasks.rados.rados.0.smithi181.stdout:289: done (0 left) 2024-04-26T01:24:55.875 INFO:tasks.rados.rados.0.smithi181.stdout:290: delete oid 41 current snap is 24 2024-04-26T01:24:55.876 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.11 to 1 (10000) 2024-04-26T01:24:55.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:24:55.876 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-26T01:24:55.877 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:55.879+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:55.906 INFO:tasks.rados.rados.0.smithi181.stdout:290: done (0 left) 2024-04-26T01:24:55.906 INFO:tasks.rados.rados.0.smithi181.stdout:291: read oid 13 snap -1 2024-04-26T01:24:55.906 INFO:tasks.rados.rados.0.smithi181.stdout:291: expect (ObjNum 91 snap 17 seq_num 91) 2024-04-26T01:24:55.906 INFO:tasks.rados.rados.0.smithi181.stdout:292: setattr oid 26 current snap is 24 2024-04-26T01:24:55.909 INFO:tasks.rados.rados.0.smithi181.stdout:293: copy_from oid 32 from oid 7 current snap is 24 2024-04-26T01:24:55.909 INFO:tasks.rados.rados.0.smithi181.stdout:294: read oid 22 snap -1 2024-04-26T01:24:55.909 INFO:tasks.rados.rados.0.smithi181.stdout:294: expect (ObjNum 108 snap 21 seq_num 108) 2024-04-26T01:24:55.909 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 15 current snap is 24 2024-04-26T01:24:55.909 INFO:tasks.rados.rados.0.smithi181.stdout:295: seq_num 113 ranges {81920=737280,819200=557056,1376256=606208} 2024-04-26T01:24:55.920 INFO:tasks.rados.rados.0.smithi181.stdout:295: writing smithi18128752-15 from 81920 to 819200 tid 1 2024-04-26T01:24:55.926 INFO:tasks.rados.rados.0.smithi181.stdout:295: writing smithi18128752-15 from 819200 to 1376256 tid 2 2024-04-26T01:24:55.932 INFO:tasks.rados.rados.0.smithi181.stdout:295: writing smithi18128752-15 from 1376256 to 1982464 tid 3 2024-04-26T01:24:55.932 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 49 current snap is 24 2024-04-26T01:24:55.932 INFO:tasks.rados.rados.0.smithi181.stdout:296: seq_num 114 ranges {819200=753664,1572864=688128,2260992=475136} 2024-04-26T01:24:55.948 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18128752-49 from 819200 to 1572864 tid 1 2024-04-26T01:24:55.956 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18128752-49 from 1572864 to 2260992 tid 2 2024-04-26T01:24:55.960 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18128752-49 from 2260992 to 2736128 tid 3 2024-04-26T01:24:55.960 INFO:tasks.rados.rados.0.smithi181.stdout:297: rollback oid 33 current snap is 24 2024-04-26T01:24:55.960 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 33 to 22 2024-04-26T01:24:55.961 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 30 current snap is 24 2024-04-26T01:24:55.961 INFO:tasks.rados.rados.0.smithi181.stdout:298: seq_num 115 ranges {5980160=770048,6750208=704512,7454720=458752} 2024-04-26T01:24:55.975 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:55.979+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:56.007 INFO:tasks.rados.rados.0.smithi181.stdout:298: writing smithi18128752-30 from 5980160 to 6750208 tid 1 2024-04-26T01:24:56.011 INFO:tasks.rados.rados.0.smithi181.stdout:298: writing smithi18128752-30 from 6750208 to 7454720 tid 2 2024-04-26T01:24:56.013 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:56.013 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:730 3.8s0 3:15703067:::smithi18128752-15:head [append 0~737280 in=737280b] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.926204+0000", 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.091885954000000006, 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091942563000000005, 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 730 2024-04-26T01:24:56.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.926204+0000", 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.926204+0000", 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.926206+0000", 2024-04-26T01:24:56.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0339999999999999e-06 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.927292+0000", 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010867089999999999 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.927313+0000", 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0264000000000001e-05 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.927321+0000", 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8589999999999994e-06 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.927401+0000", 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9637999999999999e-05 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.927504+0000", 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103284 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.928876+0000", 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001371869 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.930946+0000", 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020697210000000001 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 84/44 ECSubWriteReply(tid=140, last_complete=84'104, committed=1, applied=1))", 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.931531+0000", 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.086558204999999999, 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.086755633999999998, 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.931531+0000", 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.931531+0000", 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.931532+0000", 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.079e-06 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.931538+0000", 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6160000000000001e-06 2024-04-26T01:24:56.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.931553+0000", 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5410999999999999e-05 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.931562+0000", 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.174e-06 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:731 3.8s0 3.e60c0ea8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.932055+0000", 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.086034896, 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.086310060999999993, 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 731 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.022 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932055+0000", 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932055+0000", 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932057+0000", 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0099999999999998e-06 2024-04-26T01:24:56.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932759+0000", 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00070227600000000005 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932777+0000", 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7779999999999999e-05 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932786+0000", 2024-04-26T01:24:56.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5179999999999993e-06 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 84/44 ECSubWriteReply(tid=140, last_complete=84'104, committed=1, applied=1))", 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.932820+0000", 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.085269391999999999, 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.085632248999999994, 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932820+0000", 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932820+0000", 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932821+0000", 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9999999999999997e-07 2024-04-26T01:24:56.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932828+0000", 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8369999999999999e-06 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932838+0000", 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.927e-06 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.932843+0000", 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.631e-06 2024-04-26T01:24:56.027 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 83/44 ECSubWriteReply(tid=140, last_complete=84'104, committed=1, applied=1))", 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.934851+0000", 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.083238173999999998, 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.083675422999999999, 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.934851+0000", 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.934851+0000", 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.934852+0000", 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1100000000000005e-07 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.029 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.934857+0000", 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4839999999999998e-06 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.934870+0000", 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2979000000000001e-05 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.934876+0000", 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6050000000000003e-06 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.030 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:732 3.8s0 3.e60c0ea8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.938394+0000", 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.079695137999999999, 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.080206424999999998, 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 732 2024-04-26T01:24:56.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.938394+0000", 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.938394+0000", 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.938396+0000", 2024-04-26T01:24:56.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2729999999999999e-06 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939460+0000", 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001064621 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939492+0000", 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1362999999999998e-05 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939507+0000", 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4986e-05 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:733 3.8s0 3.e60c0ea8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.939535+0000", 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.078554641999999994, 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.079149852000000007, 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.034 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 733 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939535+0000", 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939535+0000", 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.035 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939536+0000", 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2270000000000001e-06 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939542+0000", 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5230000000000003e-06 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939554+0000", 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2150000000000001e-05 2024-04-26T01:24:56.036 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939569+0000", 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4786e-05 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:734 3.8s0 3.e60c0ea8 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.939582+0000", 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.078507335999999997, 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.079187304, 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.037 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 734 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939582+0000", 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939582+0000", 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939583+0000", 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8299999999999996e-07 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939586+0000", 2024-04-26T01:24:56.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.23e-06 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939602+0000", 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5440000000000001e-05 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.939611+0000", 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9039999999999998e-06 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.041 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.041 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:56.041 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 8 2024-04-26T01:24:56.041 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:56.043 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:298: writing smithi18128752-30 from 7454720 to 7913472 tid 3 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:299: read oid 37 snap -1 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:299: expect deleted 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:300: rollback oid 8 current snap is 24 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 8 to 23 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:301: rmattr oid 46 current snap is 24 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:301: done (10 left) 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:302: copy_from oid 3 from oid 5 current snap is 24 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 46 current snap is 24 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:303: seq_num 116 ranges {901120=786432,1687552=458752,2146304=442368,2588672=262144} 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:303: writing smithi18128752-46 from 901120 to 1687552 tid 1 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:303: writing smithi18128752-46 from 1687552 to 2146304 tid 2 2024-04-26T01:24:56.044 INFO:tasks.rados.rados.0.smithi181.stdout:303: writing smithi18128752-46 from 2146304 to 2588672 tid 3 2024-04-26T01:24:56.045 INFO:tasks.rados.rados.0.smithi181.stdout:303: writing smithi18128752-46 from 2588672 to 2850816 tid 4 2024-04-26T01:24:56.045 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 31 current snap is 24 2024-04-26T01:24:56.045 INFO:tasks.rados.rados.0.smithi181.stdout:304: seq_num 117 ranges {4276224=425984,4702208=606208,5308416=540672,5849088=409600} 2024-04-26T01:24:56.057 INFO:tasks.rados.rados.0.smithi181.stdout:304: writing smithi18128752-31 from 4276224 to 4702208 tid 1 2024-04-26T01:24:56.060 INFO:tasks.rados.rados.0.smithi181.stdout:304: writing smithi18128752-31 from 4702208 to 5308416 tid 2 2024-04-26T01:24:56.063 INFO:tasks.rados.rados.0.smithi181.stdout:304: writing smithi18128752-31 from 5308416 to 5849088 tid 3 2024-04-26T01:24:56.066 INFO:tasks.rados.rados.0.smithi181.stdout:304: writing smithi18128752-31 from 5849088 to 6258688 tid 4 2024-04-26T01:24:56.066 INFO:tasks.rados.rados.0.smithi181.stdout:305: read oid 12 snap 22 2024-04-26T01:24:56.066 INFO:tasks.rados.rados.0.smithi181.stdout:305: expect (ObjNum 106 snap 21 seq_num 106) 2024-04-26T01:24:56.066 INFO:tasks.rados.rados.0.smithi181.stdout:306: snap_create 2024-04-26T01:24:56.076 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:56.079+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:56.176 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:56.176 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:56.176 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:56.177 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:56.177 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:56.177 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:56.179+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:56.190 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:24:56.221 INFO:tasks.rados.rados.0.smithi181.stdout:302: finishing copy_from to smithi18128752-3 2024-04-26T01:24:56.222 INFO:tasks.rados.rados.0.smithi181.stdout:302: got expected ENOENT (src dne) 2024-04-26T01:24:56.222 INFO:tasks.rados.rados.0.smithi181.stdout:302: finishing copy_from racing read to smithi18128752-3 2024-04-26T01:24:56.222 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 26 v 80 (ObjNum 129 snap 0 seq_num 3032234064) dirty exists 2024-04-26T01:24:56.226 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing write tid 1 to smithi18128752-15 2024-04-26T01:24:56.278 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:56.283+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:56.331 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:129 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.755536468000003, 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.91360868200000001, 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.332 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 129 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:24:56.333 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575106+0000", 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.043e-06 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575111+0000", 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3370000000000001e-06 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.334 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575128+0000", 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7600000000000001e-05 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575138+0000", 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.877914+0000", 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30277600900000001 2024-04-26T01:24:56.335 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.878073+0000", 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015861299999999999 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979098+0000", 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101025446 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979375+0000", 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000277309 2024-04-26T01:24:56.336 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488714+0000", 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50933870599999997 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:142 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.221000463000003, 2024-04-26T01:24:56.337 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.88158457700000004, 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 142 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:56.338 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109644+0000", 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0970000000000002e-06 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.339 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110339+0000", 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00069456900000000002 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110373+0000", 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.358e-05 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110388+0000", 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5726000000000002e-05 2024-04-26T01:24:56.340 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181183+0000", 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.070795134999999995 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181419+0000", 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023538399999999999 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.182735+0000", 2024-04-26T01:24:56.341 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013160540000000001 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589098+0000", 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40636337 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991098+0000", 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401999885 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.342 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991226+0000", 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012777700000000001 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:143 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.214832800000003, 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87574223200000001, 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.343 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 143 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.344 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115811+0000", 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.317e-06 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116338+0000", 2024-04-26T01:24:56.345 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052673300000000004 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116375+0000", 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6922999999999999e-05 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116392+0000", 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6629999999999998e-05 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.346 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284095+0000", 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16770365000000001 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284265+0000", 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016969199999999999 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285568+0000", 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001302554 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.347 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589101+0000", 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30353306000000002 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991441+0000", 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40234051300000001 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991551+0000", 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011016 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.348 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:144 3.8s0 3:1fe6984d:::smithi18128752-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.214222405000001, 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87544766600000001, 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 144 2024-04-26T01:24:56.349 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.350 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116421+0000", 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.15e-06 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116427+0000", 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9070000000000001e-06 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116443+0000", 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-05 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.351 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116452+0000", 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0890000000000006e-06 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286741+0000", 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170288786 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286883+0000", 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142683 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.352 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.287610+0000", 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000727172 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589207+0000", 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30159656099999999 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991768+0000", 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40256125399999998 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.353 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991867+0000", 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8814000000000006e-05 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:145 3.8s0 3:1fe6984d:::smithi18128752-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.214176299000002, 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98038499800000001, 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.354 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 145 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:56.355 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5600000000000004e-07 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116470+0000", 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116484+0000", 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3837e-05 2024-04-26T01:24:56.356 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.216615+0000", 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100131365 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992549+0000", 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77593446499999996 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992720+0000", 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017066299999999999 2024-04-26T01:24:56.357 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096851+0000", 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104130502 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 35/30 ECSubWriteReply(tid=16, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 50.042211897999998, 2024-04-26T01:24:56.358 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60172694400000004, 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288430+0000", 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.359 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288438+0000", 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1620000000000008e-06 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288450+0000", 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1922999999999999e-05 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.288486+0000", 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5905999999999998e-05 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.360 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.388601+0000", 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10011487600000001 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890125+0000", 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50152462099999995 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.890157+0000", 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1455999999999998e-05 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.361 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:160 3.8s0 3:1945891d:::smithi18128752-34:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.826026298000002, 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.79222307800000002, 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 160 2024-04-26T01:24:56.362 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504616+0000", 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504618+0000", 2024-04-26T01:24:56.363 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5600000000000001e-06 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504629+0000", 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0956e-05 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504661+0000", 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2428999999999997e-05 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.504678+0000", 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6422999999999998e-05 2024-04-26T01:24:56.364 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093363+0000", 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58868559200000004 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.093514+0000", 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015052300000000001 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.094369+0000", 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00085476900000000001 2024-04-26T01:24:56.365 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096327+0000", 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001957781 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296743+0000", 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20041606200000001 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.296839+0000", 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5983000000000006e-05 2024-04-26T01:24:56.366 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:269 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 44.156920134000003, 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81822368899999998, 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.367 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 269 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.368 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173723+0000", 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4399999999999999e-07 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173738+0000", 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5798999999999999e-05 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173754+0000", 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5943e-05 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.369 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173763+0000", 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1730000000000001e-06 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385572+0000", 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.211808092 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385724+0000", 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015250999999999999 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.370 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589244+0000", 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203520116 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589408+0000", 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016355699999999999 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.991945+0000", 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40253765499999999 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.371 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:383 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~720896] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.109894591, 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50440791100000004, 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.372 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 383 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220747+0000", 2024-04-26T01:24:56.373 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.220749+0000", 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.995e-06 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221593+0000", 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00084381899999999999 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.374 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221614+0000", 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1061999999999999e-05 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221623+0000", 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.3130000000000004e-06 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.317149+0000", 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.095525231000000002 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.375 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.317292+0000", 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014299199999999999 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318544+0000", 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001252567 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322759+0000", 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042146029999999999 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.376 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.724936+0000", 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40217761699999999 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725155+0000", 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000218712 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.377 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:384 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~49152] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.108998395999997, 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50390550300000003, 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.378 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 384 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221643+0000", 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.379 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221644+0000", 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.0200000000000001e-07 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221675+0000", 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0929999999999997e-05 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221683+0000", 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3559999999999993e-06 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.380 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221689+0000", 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7370000000000003e-06 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.319416+0000", 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.097727075999999996 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.319496+0000", 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9579000000000005e-05 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.381 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.320065+0000", 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00056965599999999998 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322760+0000", 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026947189999999999 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725421+0000", 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40266085000000001 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.382 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725549+0000", 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012789799999999999 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:385 3.8s0 3:14c0c421:::smithi18128752-31:head [setxattr _header (56) in=7b] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.108942874999997, 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50445409900000004, 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.383 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 385 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.384 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221699+0000", 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.82e-07 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.385 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221702+0000", 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.779e-06 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221715+0000", 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2337000000000001e-05 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.386 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221721+0000", 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.207e-06 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.321880+0000", 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100158994 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.321983+0000", 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103669 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.387 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.322488+0000", 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050421999999999999 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.423268+0000", 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10077992299999999 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.725960+0000", 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30269248500000001 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.388 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.726153+0000", 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000193003 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:386 3.8s0 3:14c0c421:::smithi18128752-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.108912897000003, 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70800976199999999, 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.389 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 386 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.390 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221729+0000", 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221732+0000", 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1129999999999999e-06 2024-04-26T01:24:56.391 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221738+0000", 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4609999999999996e-06 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.221744+0000", 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3539999999999996e-06 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727164+0000", 2024-04-26T01:24:56.392 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50541931600000001 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727250+0000", 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6618000000000003e-05 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.929739+0000", 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20248843399999999 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.393 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:394 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.090256390999997, 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.79330826399999999, 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 394 2024-04-26T01:24:56.394 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240385+0000", 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.395 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240388+0000", 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3410000000000001e-06 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240398+0000", 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8570000000000007e-06 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240421+0000", 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3198999999999999e-05 2024-04-26T01:24:56.396 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240451+0000", 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.023e-05 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727765+0000", 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.487313618 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.727862+0000", 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7324999999999994e-05 2024-04-26T01:24:56.397 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.033694+0000", 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30583169399999999 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:395 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.090168153, 2024-04-26T01:24:56.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.89472404000000005, 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 395 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4199999999999995e-07 2024-04-26T01:24:56.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240479+0000", 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8199999999999996e-06 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240489+0000", 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7089999999999997e-06 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240496+0000", 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9920000000000001e-06 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829732+0000", 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58923568800000004 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829882+0000", 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015056100000000001 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.135198+0000", 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.305315528 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:396 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:56.403 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.090111812000004, 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99608940400000001, 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 396 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240531+0000", 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0800000000000004e-07 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240536+0000", 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2059999999999998e-06 2024-04-26T01:24:56.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240547+0000", 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1663e-05 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240555+0000", 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1010000000000003e-06 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831852+0000", 2024-04-26T01:24:56.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.59129766500000003 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832019+0000", 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166773 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.236619+0000", 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40460038799999998 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:606 3.8s0 3:18914e2a:::smithi18128752-43:head [append 0~524288] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.796417+0000", 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.534225142, 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77806377400000004, 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 606 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.796417+0000", 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.796417+0000", 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.796420+0000", 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.066e-06 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.797625+0000", 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012048880000000001 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.797659+0000", 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4703000000000003e-05 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.797676+0000", 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6670000000000001e-05 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.823878+0000", 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026201616 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op must wait for map", 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.823899+0000", 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1305999999999999e-05 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.925988+0000", 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10208885199999999 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.926215+0000", 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00022680199999999999 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.928006+0000", 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017915050000000001 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.936485+0000", 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0084790630000000002 2024-04-26T01:24:56.412 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.574345+0000", 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.63786011899999995 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.574480+0000", 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000135184 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:607 3.8s0 3:18914e2a:::smithi18128752-43:head [append 0~753664] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:56.413 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.802289+0000", 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.528353055, 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.772481431, 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 607 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.802289+0000", 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.802289+0000", 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.802291+0000", 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2230000000000001e-06 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.803776+0000", 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001484576 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.803809+0000", 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3152999999999999e-05 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.803824+0000", 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5719e-05 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.824320+0000", 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.020495658999999999 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.824339+0000", 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9119000000000001e-05 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.929570+0000", 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105230423 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.929870+0000", 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030064000000000002 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.932067+0000", 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021965539999999999 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.418 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.936488+0000", 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0044212110000000004 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.574662+0000", 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.638174307 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.574770+0000", 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000107847 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:608 3.8s0 3:18914e2a:::smithi18128752-43:head [append 0~409600] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.805377+0000", 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.525265263, 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87196500700000001, 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.423 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 608 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.805377+0000", 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.805377+0000", 2024-04-26T01:24:56.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.805379+0000", 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2900000000000001e-06 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806136+0000", 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000757601 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806169+0000", 2024-04-26T01:24:56.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2316e-05 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806184+0000", 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5546e-05 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.897941+0000", 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091756901000000002 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.897953+0000", 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.217e-05 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.933446+0000", 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.035492194999999997 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.933600+0000", 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015479 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.934868+0000", 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012679200000000001 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.938268+0000", 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033999490000000002 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.677202+0000", 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.73893376499999996 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.677342+0000", 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013956400000000001 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:609 3.8s0 3:18914e2a:::smithi18128752-43:head [setxattr _header (57) in=7b] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.806213+0000", 2024-04-26T01:24:56.429 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.524428834, 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97183218800000004, 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 609 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806213+0000", 2024-04-26T01:24:56.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806213+0000", 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806214+0000", 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4300000000000001e-06 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806225+0000", 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0906e-05 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806246+0000", 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0276999999999999e-05 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806261+0000", 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5136e-05 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.898005+0000", 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091744395000000006 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:56.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.898014+0000", 2024-04-26T01:24:56.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4460000000000001e-06 2024-04-26T01:24:56.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.936586+0000", 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038572137999999999 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.936741+0000", 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015529000000000001 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.937477+0000", 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00073645700000000004 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.940385+0000", 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029071230000000002 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.777805+0000", 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.83742000500000002 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.778045+0000", 2024-04-26T01:24:56.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000240585 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:610 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.806287+0000", 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.524355121999999, 2024-04-26T01:24:56.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0756521729999999, 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 610 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.439 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806287+0000", 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806287+0000", 2024-04-26T01:24:56.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806288+0000", 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4869999999999999e-06 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806295+0000", 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4359999999999999e-06 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806320+0000", 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5343000000000001e-05 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806335+0000", 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4895999999999999e-05 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911536+0000", 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105201143 2024-04-26T01:24:56.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911557+0000", 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0933000000000001e-05 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.938102+0000", 2024-04-26T01:24:56.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026545052 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.938251+0000", 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014903099999999999 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:56.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.778133+0000", 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.83988231599999996 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.778287+0000", 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015381699999999999 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.881939+0000", 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103651719 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.447 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:56.449 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing write tid 2 to smithi18128752-15 2024-04-26T01:24:56.449 INFO:tasks.rados.rados.0.smithi181.stdout:293: finishing copy_from racing read to smithi18128752-32 2024-04-26T01:24:56.449 INFO:tasks.rados.rados.0.smithi181.stdout:291: expect (ObjNum 91 snap 17 seq_num 91) 2024-04-26T01:24:56.449 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:56.452+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:56.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:24:56.549 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:56.551+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:56.585 INFO:tasks.rados.rados.0.smithi181.stdout:305: expect (ObjNum 106 snap 21 seq_num 106) 2024-04-26T01:24:56.628 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:56.629 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:56.629 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.629 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.4s1 84/44 ECSubWrite(tid=82, reqid=client.4589.0:746, at_version=0'0, trim_to=0'0, roll_forward_to=84'49))", 2024-04-26T01:24:56.629 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.630826+0000", 2024-04-26T01:24:56.629 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.0025512310000000002, 2024-04-26T01:24:56.629 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002569825, 2024-04-26T01:24:56.629 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.630 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.630 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:24:56.630 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.630 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.630 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.630 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.630826+0000", 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.630826+0000", 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.631 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.630828+0000", 2024-04-26T01:24:56.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.973e-06 2024-04-26T01:24:56.632 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.632 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.632 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.632 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.630856+0000", 2024-04-26T01:24:56.632 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7546999999999999e-05 2024-04-26T01:24:56.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.633 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.633 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.630898+0000", 2024-04-26T01:24:56.633 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.2199e-05 2024-04-26T01:24:56.633 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.633 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.634 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:56.634 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.630921+0000", 2024-04-26T01:24:56.634 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3493e-05 2024-04-26T01:24:56.634 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.634 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.634 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.635 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.635 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.635 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.4s1 84/44 ECSubRead(tid=83, to_read={3:2be1e129:::smithi18128752-30:head=0,4096,0}, subchunks={3:2be1e129:::smithi18128752-30:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:56.635 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.632074+0000", 2024-04-26T01:24:56.635 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.001303472, 2024-04-26T01:24:56.635 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001356643, 2024-04-26T01:24:56.636 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:56.636 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:56.636 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "no flag points reached", 2024-04-26T01:24:56.636 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:56.636 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.636 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:56.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.632074+0000", 2024-04-26T01:24:56.637 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.637 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.637 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.637 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:56.637 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.632074+0000", 2024-04-26T01:24:56.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:56.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.638 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.638 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:56.638 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.632076+0000", 2024-04-26T01:24:56.638 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7799999999999999e-06 2024-04-26T01:24:56.638 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.639 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:56.639 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.632094+0000", 2024-04-26T01:24:56.639 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8660000000000001e-05 2024-04-26T01:24:56.639 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:56.639 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:56.640 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:56.640 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.632133+0000", 2024-04-26T01:24:56.640 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.9054999999999998e-05 2024-04-26T01:24:56.640 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.640 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:56.640 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.641 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:56.641 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:24:56.641 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 2 2024-04-26T01:24:56.641 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:56.644 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:24:56.650 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:56.655+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:56.695 INFO:tasks.rados.rados.0.smithi181.stdout:297: finishing rollback tid 0 to smithi18128752-33 2024-04-26T01:24:56.695 INFO:tasks.rados.rados.0.smithi181.stdout:294: expect (ObjNum 108 snap 21 seq_num 108) 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 1 to smithi18128752-49 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 2 to smithi18128752-49 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:303: finishing write tid 1 to smithi18128752-46 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:293: finishing copy_from to smithi18128752-32 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 32 v 11 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:297: finishing rollback tid 1 to smithi18128752-33 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:297: finishing rollback tid 2 to smithi18128752-33 2024-04-26T01:24:56.741 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 70 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:298: finishing write tid 1 to smithi18128752-30 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:298: finishing write tid 2 to smithi18128752-30 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:298: finishing write tid 3 to smithi18128752-30 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:298: finishing write tid 4 to smithi18128752-30 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 3 to smithi18128752-49 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:303: finishing write tid 2 to smithi18128752-46 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:303: finishing write tid 3 to smithi18128752-46 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 4 to smithi18128752-49 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing write tid 3 to smithi18128752-15 2024-04-26T01:24:56.742 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing write tid 4 to smithi18128752-15 2024-04-26T01:24:56.751 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:56.755+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:56.809 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:56.809 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:56.809 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:56.809 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:56.809 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:56.823 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:24:56.830 INFO:tasks.rados.rados.0.smithi181.stdout:300: finishing rollback tid 0 to smithi18128752-8 2024-04-26T01:24:56.830 INFO:tasks.rados.rados.0.smithi181.stdout:300: finishing rollback tid 1 to smithi18128752-8 2024-04-26T01:24:56.831 INFO:tasks.rados.rados.0.smithi181.stdout:303: finishing write tid 4 to smithi18128752-46 2024-04-26T01:24:56.835 INFO:tasks.rados.rados.0.smithi181.stdout:300: finishing rollback tid 2 to smithi18128752-8 2024-04-26T01:24:56.836 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 134 (ObjNum 69 snap 6 seq_num 69) dirty exists 2024-04-26T01:24:56.866 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:56.871+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:56.879 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:24:56.886 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:24:56.930 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 5 to smithi18128752-49 2024-04-26T01:24:56.930 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 49 updating version 0 to 71 2024-04-26T01:24:56.930 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 49 version 71 is already newer than 68 2024-04-26T01:24:56.930 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 49 version 71 is already newer than 69 2024-04-26T01:24:56.930 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 49 version 71 is already newer than 70 2024-04-26T01:24:56.930 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 49 v 71 (ObjNum 114 snap 24 seq_num 114) dirty exists 2024-04-26T01:24:56.930 INFO:tasks.rados.rados.0.smithi181.stdout:296: left oid 49 (ObjNum 114 snap 24 seq_num 114) 2024-04-26T01:24:56.931 INFO:tasks.rados.rados.0.smithi181.stdout:303: finishing write tid 5 to smithi18128752-46 2024-04-26T01:24:56.938 INFO:tasks.rados.rados.0.smithi181.stdout:303: finishing write tid 6 to smithi18128752-46 2024-04-26T01:24:56.938 INFO:tasks.rados.rados.0.smithi181.stdout:303: oid 46 updating version 0 to 41 2024-04-26T01:24:56.939 INFO:tasks.rados.rados.0.smithi181.stdout:303: oid 46 version 41 is already newer than 38 2024-04-26T01:24:56.939 INFO:tasks.rados.rados.0.smithi181.stdout:303: oid 46 version 41 is already newer than 39 2024-04-26T01:24:56.939 INFO:tasks.rados.rados.0.smithi181.stdout:303: oid 46 version 41 is already newer than 37 2024-04-26T01:24:56.939 INFO:tasks.rados.rados.0.smithi181.stdout:303: oid 46 version 41 is already newer than 40 2024-04-26T01:24:56.939 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 46 v 41 (ObjNum 116 snap 24 seq_num 116) dirty exists 2024-04-26T01:24:56.939 INFO:tasks.rados.rados.0.smithi181.stdout:303: left oid 46 (ObjNum 116 snap 24 seq_num 116) 2024-04-26T01:24:56.940 INFO:tasks.rados.rados.0.smithi181.stdout:304: finishing write tid 1 to smithi18128752-31 2024-04-26T01:24:56.940 INFO:tasks.rados.rados.0.smithi181.stdout:304: finishing write tid 2 to smithi18128752-31 2024-04-26T01:24:56.940 INFO:tasks.rados.rados.0.smithi181.stdout:304: finishing write tid 3 to smithi18128752-31 2024-04-26T01:24:56.940 INFO:tasks.rados.rados.0.smithi181.stdout:304: finishing write tid 4 to smithi18128752-31 2024-04-26T01:24:56.940 INFO:tasks.rados.rados.0.smithi181.stdout:304: finishing write tid 5 to smithi18128752-31 2024-04-26T01:24:56.941 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.13 to 1 (10000) 2024-04-26T01:24:56.948 INFO:tasks.rados.rados.0.smithi181.stdout:291: done (14 left) 2024-04-26T01:24:56.948 INFO:tasks.rados.rados.0.smithi181.stdout:292: done (13 left) 2024-04-26T01:24:56.948 INFO:tasks.rados.rados.0.smithi181.stdout:293: done (12 left) 2024-04-26T01:24:56.948 INFO:tasks.rados.rados.0.smithi181.stdout:294: done (11 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:296: done (10 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:297: done (9 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:299: done (8 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:300: done (7 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:302: done (6 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:303: done (5 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:305: done (4 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:306: done (3 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:307: rmattr oid 39 current snap is 25 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:307: done (3 left) 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:308: copy_from oid 4 from oid 13 current snap is 25 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:309: rollback oid 2 current snap is 25 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 2 to 23 2024-04-26T01:24:56.949 INFO:tasks.rados.rados.0.smithi181.stdout:310: snap_remove snap 24 2024-04-26T01:24:56.953 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:24:56.955+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:24:56.971 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-26T01:24:57.003 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:57.003 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s6 37/34 ECSubWrite(tid=16, reqid=client.4589.0:181, at_version=37'9, trim_to=0'0, roll_forward_to=36'8))", 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.933914367, 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20273409100000001, 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.004 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.071886+0000", 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:24:57.005 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.072132+0000", 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024541999999999999 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.072149+0000", 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7192000000000001e-05 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.006 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172254+0000", 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100104756 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172426+0000", 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017276600000000001 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.172443+0000", 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7136e-05 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.007 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.274572+0000", 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102128393 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:08.274620+0000", 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8152999999999998e-05 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 41/34 ECSubWrite(tid=46, reqid=osd.2.0:46, at_version=41'29, trim_to=0'0, roll_forward_to=41'27))", 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.599843+0000", 2024-04-26T01:24:57.008 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 44.405957778999998, 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20203437599999999, 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599843+0000", 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599843+0000", 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.009 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599844+0000", 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6840000000000001e-06 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599881+0000", 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6418999999999998e-05 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599908+0000", 2024-04-26T01:24:57.010 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7053e-05 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.599925+0000", 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7278999999999998e-05 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800207+0000", 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200281828 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.800221+0000", 2024-04-26T01:24:57.011 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4039999999999999e-05 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.801788+0000", 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015674650000000001 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.801877+0000", 2024-04-26T01:24:57.012 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8608000000000003e-05 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.bs3 61/34 ECSubWrite(tid=175, reqid=client.4589.0:473, at_version=61'12, trim_to=0'0, roll_forward_to=48'10))", 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.883901+0000", 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 24.121899020000001, 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.406334209, 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.013 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.883901+0000", 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.883901+0000", 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.014 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.883903+0000", 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6530000000000001e-06 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.884544+0000", 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064116599999999998 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.884570+0000", 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6146e-05 2024-04-26T01:24:57.015 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.884587+0000", 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6803000000000002e-05 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.884697+0000", 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110086 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.884709+0000", 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1853e-05 2024-04-26T01:24:57.016 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.290123+0000", 2024-04-26T01:24:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40541378099999997 2024-04-26T01:24:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.017 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.290236+0000", 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011272099999999999 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.bs3 61/34 ECSubWrite(tid=179, reqid=client.4589.0:474, at_version=61'13, trim_to=0'0, roll_forward_to=48'10))", 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.989904+0000", 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 24.015896653999999, 2024-04-26T01:24:57.018 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30046578499999999, 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.989904+0000", 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.989904+0000", 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.019 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.989906+0000", 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9300000000000002e-06 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.990263+0000", 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00035737099999999999 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.020 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.990287+0000", 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3683e-05 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.990309+0000", 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2204999999999998e-05 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.187798+0000", 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.197489479 2024-04-26T01:24:57.021 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.187814+0000", 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5226e-05 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.290296+0000", 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10248220299999999 2024-04-26T01:24:57.022 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.290369+0000", 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3688000000000003e-05 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.bs3 61/34 ECSubWrite(tid=180, reqid=client.4589.0:475, at_version=61'14, trim_to=0'0, roll_forward_to=48'10))", 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.991323+0000", 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 24.014477438, 2024-04-26T01:24:57.023 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.299142397, 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.991323+0000", 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.991323+0000", 2024-04-26T01:24:57.024 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.991324+0000", 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.39e-06 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.991401+0000", 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6613999999999995e-05 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.025 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.991428+0000", 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7529e-05 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.991444+0000", 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5194000000000001e-05 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.026 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.188907+0000", 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.197463209 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.188923+0000", 2024-04-26T01:24:57.027 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6163000000000001e-05 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.290407+0000", 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101484136 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.290465+0000", 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8162000000000003e-05 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.028 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.bs3 61/34 ECSubWrite(tid=181, reqid=client.4589.0:476, at_version=61'15, trim_to=0'0, roll_forward_to=48'10))", 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.992136+0000", 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 24.013663944000001, 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29929788099999999, 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.029 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.992136+0000", 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.992136+0000", 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.030 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.992138+0000", 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4169999999999999e-06 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.992176+0000", 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8219000000000001e-05 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.031 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.992201+0000", 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5205999999999999e-05 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.092310+0000", 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100108867 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.032 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.289720+0000", 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19741032 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.289736+0000", 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5532999999999998e-05 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.033 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.291361+0000", 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016250780000000001 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.291434+0000", 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3240999999999999e-05 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.034 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 61/44 ECSubWrite(tid=240, reqid=client.4589.0:484, at_version=61'44, trim_to=0'0, roll_forward_to=48'40))", 2024-04-26T01:24:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:33.005133+0000", 2024-04-26T01:24:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 24.000667672999999, 2024-04-26T01:24:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20168561400000001, 2024-04-26T01:24:57.035 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.036 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.036 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.036 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.036 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.005133+0000", 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.005133+0000", 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.037 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.005135+0000", 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1169999999999998e-06 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.005319+0000", 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018436200000000001 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.038 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.005360+0000", 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1152000000000002e-05 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.105457+0000", 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10009699 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.105568+0000", 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000110967 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.039 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.105583+0000", 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5057e-05 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.206760+0000", 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10117643799999999 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.206818+0000", 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8530999999999999e-05 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.040 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.as3 65/63 ECSubRead(tid=24, to_read={3:56e8b2fd:::smithi18128752-12:head=0,4096,0}, subchunks={3:56e8b2fd:::smithi18128752-12:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:37.944647+0000", 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 19.061153295, 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20096275699999999, 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.041 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:37.944647+0000", 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:37.944647+0000", 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:37.944648+0000", 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2e-07 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.042 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:37.944663+0000", 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5486999999999999e-05 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:37.944700+0000", 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6504999999999997e-05 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.044842+0000", 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10014290300000001 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.043 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.145251+0000", 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10040837399999999 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.145610+0000", 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00035896799999999998 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.044 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 67/44 ECSubWrite(tid=108, reqid=osd.5.0:108, at_version=67'76, trim_to=0'0, roll_forward_to=65'72))", 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.785759+0000", 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.220041813000002, 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30440253699999997, 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.785759+0000", 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.045 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.785759+0000", 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.785760+0000", 2024-04-26T01:24:57.046 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7540000000000001e-06 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.785818+0000", 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7272000000000003e-05 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.785856+0000", 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7988999999999998e-05 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.785881+0000", 2024-04-26T01:24:57.047 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5168999999999999e-05 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.886803+0000", 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100922321 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.886819+0000", 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6013000000000001e-05 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.090117+0000", 2024-04-26T01:24:57.048 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20329761299999999 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.090161+0000", 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4406000000000001e-05 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s3 67/44 ECSubRead(tid=110, to_read={3:1fe6984d:::smithi18128752-48:head=0,811008,0}, subchunks={3:1fe6984d:::smithi18128752-48:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.796214+0000", 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.209586285, 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29389491099999998, 2024-04-26T01:24:57.049 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.796214+0000", 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.796214+0000", 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.050 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.796216+0000", 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2000000000000001e-06 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.796239+0000", 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2626e-05 2024-04-26T01:24:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.796290+0000", 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1221999999999998e-05 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.052 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.796321+0000", 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0828999999999999e-05 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.089681+0000", 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29336035599999999 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.053 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.090109+0000", 2024-04-26T01:24:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00042767800000000002 2024-04-26T01:24:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.054 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s3 67/44 ECSubRead(tid=111, to_read={3:1fe6984d:::smithi18128752-48:head=0,811008,0}, subchunks={3:1fe6984d:::smithi18128752-48:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.797509+0000", 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.208290995999999, 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29327033600000002, 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.055 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797509+0000", 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797509+0000", 2024-04-26T01:24:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797511+0000", 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6899999999999999e-06 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797530+0000", 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9378999999999999e-05 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797564+0000", 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3878999999999999e-05 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.058 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797593+0000", 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8566999999999999e-05 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.090287+0000", 2024-04-26T01:24:57.059 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.292694389 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.090780+0000", 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000492432 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.060 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.1s2 67/44 ECSubWrite(tid=194, reqid=osd.4.0:194, at_version=67'16, trim_to=0'0, roll_forward_to=45'14))", 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.929762+0000", 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 18.076038782000001, 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20704893599999999, 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.061 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.929762+0000", 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.929762+0000", 2024-04-26T01:24:57.062 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.929762+0000", 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.7000000000000005e-07 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.063 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.929807+0000", 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.5349000000000002e-05 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.929853+0000", 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6013000000000001e-05 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.064 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.929879+0000", 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5972000000000001e-05 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.130004+0000", 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200124311 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.065 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.130020+0000", 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6633000000000002e-05 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.136723+0000", 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0067029799999999999 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.136810+0000", 2024-04-26T01:24:57.066 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7107999999999993e-05 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.as3 67/63 ECSubWrite(tid=28, reqid=osd.3.0:27, at_version=0'0, trim_to=0'0, roll_forward_to=67'23))", 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.197182+0000", 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.808618043999999, 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20170823700000001, 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.067 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.197182+0000", 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.197182+0000", 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.068 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.197183+0000", 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7599999999999996e-07 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.197199+0000", 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6274e-05 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.197254+0000", 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5121000000000003e-05 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.069 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.197276+0000", 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1783000000000001e-05 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.397626+0000", 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20034949199999999 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.397643+0000", 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.73e-05 2024-04-26T01:24:57.070 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.398778+0000", 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0011346170000000001 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.398891+0000", 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112874 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.071 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.7s2 67/30 ECSubWrite(tid=207, reqid=osd.4.0:207, at_version=67'50, trim_to=0'0, roll_forward_to=67'46))", 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.339909+0000", 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.665891579, 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20183847299999999, 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.072 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.339909+0000", 2024-04-26T01:24:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.339909+0000", 2024-04-26T01:24:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.073 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.339910+0000", 2024-04-26T01:24:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4900000000000004e-07 2024-04-26T01:24:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.074 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.339932+0000", 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2574000000000001e-05 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.339949+0000", 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6277999999999999e-05 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.440032+0000", 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100083478 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.075 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.540247+0000", 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10021508799999999 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.540298+0000", 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1381999999999997e-05 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.541684+0000", 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013859320000000001 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.076 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.541747+0000", 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2792000000000005e-05 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 67/44 ECSubWrite(tid=116, reqid=osd.5.0:114, at_version=0'0, trim_to=0'0, roll_forward_to=67'80))", 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.602876+0000", 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 17.402924106, 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20287972500000001, 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.077 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.602876+0000", 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.602876+0000", 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.602877+0000", 2024-04-26T01:24:57.078 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.11e-06 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.602889+0000", 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1837e-05 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.602903+0000", 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3913e-05 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.079 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.602930+0000", 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6458999999999999e-05 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.703167+0000", 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10023722 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.703200+0000", 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2978999999999997e-05 2024-04-26T01:24:57.080 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.805606+0000", 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102406071 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.805756+0000", 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015013599999999999 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.081 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.4s1 72/44 ECSubWrite(tid=44, reqid=client.4589.0:611, at_version=72'19, trim_to=0'0, roll_forward_to=71'18))", 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.916730+0000", 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 13.089070482, 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200908381, 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.916730+0000", 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.082 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.916730+0000", 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.916730+0000", 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.0999999999999999e-07 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.917154+0000", 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00042399499999999998 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.083 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.917181+0000", 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6752000000000001e-05 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.917196+0000", 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5126999999999999e-05 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.017519+0000", 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100323182 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.084 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.017548+0000", 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8218999999999999e-05 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.117558+0000", 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10001054099999999 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.117638+0000", 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0055000000000005e-05 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.085 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.7s2 72/30 ECSubRead(tid=261, to_read={3:e977e128:::smithi18128752-21:head=0,1036288,0}, subchunks={3:e977e128:::smithi18128752-21:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:44.232903+0000", 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.772896894000001, 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20226486900000001, 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.086 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.232903+0000", 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.232903+0000", 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.232905+0000", 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.776e-06 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.087 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.232923+0000", 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8267e-05 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.232957+0000", 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3556000000000002e-05 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.333053+0000", 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100096163 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.088 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.433347+0000", 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10029373499999999 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.435168+0000", 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0018213719999999999 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 79/44 ECSubWrite(tid=137, reqid=osd.5.0:136, at_version=0'0, trim_to=0'0, roll_forward_to=79'100))", 2024-04-26T01:24:57.089 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:51.321165+0000", 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 5.684635203, 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20211641, 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321165+0000", 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321165+0000", 2024-04-26T01:24:57.090 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321167+0000", 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3530000000000002e-06 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.091 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321196+0000", 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8725999999999999e-05 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321238+0000", 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1406000000000003e-05 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.092 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321263+0000", 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4939999999999998e-05 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321453+0000", 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019082999999999999 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.321484+0000", 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0568999999999998e-05 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.093 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.523189+0000", 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20170495399999999 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.523282+0000", 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2632000000000004e-05 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.094 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es1 84/44 ECSubWrite(tid=309, reqid=client.4589.0:735, at_version=84'68, trim_to=0'0, roll_forward_to=78'66))", 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.019258+0000", 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.98654247900000003, 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20300006900000001, 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.019258+0000", 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.095 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.019258+0000", 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.019260+0000", 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9759999999999998e-06 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.019605+0000", 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00034476500000000002 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.096 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.019642+0000", 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7178999999999998e-05 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.019664+0000", 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.251e-05 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.220188+0000", 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200523601 2024-04-26T01:24:57.097 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.220203+0000", 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5571000000000001e-05 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.222227+0000", 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002023108 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.098 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.222258+0000", 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1359000000000002e-05 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 84/44 ECSubWrite(tid=141, reqid=client.4589.0:731, at_version=84'105, trim_to=0'0, roll_forward_to=82'102))", 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.034921+0000", 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.97087969600000001, 2024-04-26T01:24:57.099 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20105524899999999, 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.034921+0000", 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:24:57.100 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.034921+0000", 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.034923+0000", 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3829999999999999e-06 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.035128+0000", 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020482000000000001 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.101 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.035157+0000", 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8782e-05 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.135265+0000", 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100108172 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.135330+0000", 2024-04-26T01:24:57.102 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5036000000000005e-05 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.135338+0000", 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4889999999999995e-06 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.235932+0000", 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100593707 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.235976+0000", 2024-04-26T01:24:57.103 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3859999999999997e-05 2024-04-26T01:24:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:24:57.104 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:24:57.104 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:57.106 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:57.106 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:57.107 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:57.107 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:57.108 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:24:57.108 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:57.108 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:57.110 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:57.108+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:57.137 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:57.137 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:57.170 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:57.171 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:57.207 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:57.208+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:57.242 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:57.242 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:57.242 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-04-26T01:24:57.243 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:57.247 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:57.247 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:57.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:24:57.281 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:57.282 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:57.302 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:57.302 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:57.306 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:57.311+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:57.332 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:57.332 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:57.352 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:57.353 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:57.374 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:57.375 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:57.385 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:24:57.385 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:24:57.385 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:24:57.386 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:24:57.386 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:24:57.396 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:57.396 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:57.400 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:24:57.408 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:57.412+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:57.415 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:57.416 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:57.435 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:57.436 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:57.459 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:24:57.459 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:24:57.481 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:24:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:24:57.508 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:57.511+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:57.522 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:57.522 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:57.536 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:24:57.609 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:57.611+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:57.667 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:57.667 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:57.682 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:24:57.710 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:24:57.715+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:24:57.805 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:57.806 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:57.811 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:57.815+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:57.820 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:24:57.912 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:57.916+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:57.942 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:57.943 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:57.957 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:24:57.958 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-04-26T01:24:57.958 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.14 to 1 (10000) 2024-04-26T01:24:57.968 INFO:tasks.rados.rados.0.smithi181.stdout:310: done (5 left) 2024-04-26T01:24:57.968 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 25 2024-04-26T01:24:57.968 INFO:tasks.rados.rados.0.smithi181.stdout:311: seq_num 118 ranges {1458176=442368,1900544=753664,2654208=655360,3309568=147456} 2024-04-26T01:24:57.978 INFO:tasks.rados.rados.0.smithi181.stdout:311: writing smithi18128752-50 from 1458176 to 1900544 tid 1 2024-04-26T01:24:57.983 INFO:tasks.rados.rados.0.smithi181.stdout:311: writing smithi18128752-50 from 1900544 to 2654208 tid 2 2024-04-26T01:24:57.986 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-26T01:24:57.988 INFO:tasks.rados.rados.0.smithi181.stdout:311: writing smithi18128752-50 from 2654208 to 3309568 tid 3 2024-04-26T01:24:57.989 INFO:tasks.rados.rados.0.smithi181.stdout:311: writing smithi18128752-50 from 3309568 to 3457024 tid 4 2024-04-26T01:24:57.989 INFO:tasks.rados.rados.0.smithi181.stdout:312: snap_remove snap 22 2024-04-26T01:24:58.013 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:58.015+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:58.085 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:58.085 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:58.103 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:24:58.148 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:24:58.151+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:24:58.214 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:24:58.215+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:24:58.221 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:58.221 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:58.235 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:24:58.315 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:24:58.316+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:24:58.396 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:58.396 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:58.410 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:24:58.416 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:24:58.419+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:24:58.517 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:24:58.519+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:24:58.534 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:58.534 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:58.549 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:24:58.618 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:24:58.619+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:24:58.666 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:58.666 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:58.680 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:24:58.719 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:58.723+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:58.778 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.15 to 1 (10000) 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:312: done (6 left) 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:313: rollback oid 46 current snap is 25 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 46 to 23 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:298: finishing write tid 5 to smithi18128752-30 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:298: oid 30 updating version 0 to 72 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:298: oid 30 updating version 72 to 74 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:298: oid 30 version 74 is already newer than 73 2024-04-26T01:24:58.785 INFO:tasks.rados.rados.0.smithi181.stdout:298: oid 30 version 74 is already newer than 71 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 74 (ObjNum 115 snap 24 seq_num 115) dirty exists 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:298: left oid 30 (ObjNum 115 snap 24 seq_num 115) 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:309: finishing rollback tid 1 to smithi18128752-2 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 0 (ObjNum 33 snap 0 seq_num 1550935912) dirty dne 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:308: finishing copy_from racing read to smithi18128752-4 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:304: finishing write tid 6 to smithi18128752-31 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:304: oid 31 updating version 0 to 110 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:304: oid 31 version 110 is already newer than 109 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:304: oid 31 updating version 110 to 111 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:304: oid 31 updating version 111 to 112 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:304: oid 31 updating version 112 to 113 2024-04-26T01:24:58.786 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 113 (ObjNum 117 snap 24 seq_num 117) dirty exists 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:304: left oid 31 (ObjNum 117 snap 24 seq_num 117) 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing write tid 5 to smithi18128752-15 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:295: oid 15 updating version 0 to 104 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:295: oid 15 updating version 104 to 105 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:295: oid 15 updating version 105 to 107 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:295: oid 15 version 107 is already newer than 106 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 107 (ObjNum 113 snap 24 seq_num 113) dirty exists 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:295: left oid 15 (ObjNum 113 snap 24 seq_num 113) 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:308: finishing copy_from to smithi18128752-4 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 108 (ObjNum 91 snap 17 seq_num 91) dirty exists 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:311: finishing write tid 1 to smithi18128752-50 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:311: finishing write tid 2 to smithi18128752-50 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:295: done (6 left) 2024-04-26T01:24:58.787 INFO:tasks.rados.rados.0.smithi181.stdout:298: done (5 left) 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:304: done (4 left) 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:308: done (3 left) 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:309: done (2 left) 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:314: setattr oid 15 current snap is 25 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:311: finishing write tid 3 to smithi18128752-50 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:311: finishing write tid 4 to smithi18128752-50 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:311: finishing write tid 5 to smithi18128752-50 2024-04-26T01:24:58.788 INFO:tasks.rados.rados.0.smithi181.stdout:315: snap_create 2024-04-26T01:24:58.810 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-26T01:24:58.811 INFO:tasks.rados.rados.0.smithi181.stdout:311: finishing write tid 6 to smithi18128752-50 2024-04-26T01:24:58.811 INFO:tasks.rados.rados.0.smithi181.stdout:311: oid 50 updating version 0 to 84 2024-04-26T01:24:58.811 INFO:tasks.rados.rados.0.smithi181.stdout:311: oid 50 updating version 84 to 86 2024-04-26T01:24:58.811 INFO:tasks.rados.rados.0.smithi181.stdout:311: oid 50 version 86 is already newer than 85 2024-04-26T01:24:58.811 INFO:tasks.rados.rados.0.smithi181.stdout:311: oid 50 version 86 is already newer than 83 2024-04-26T01:24:58.811 INFO:tasks.rados.rados.0.smithi181.stdout:311: oid 50 version 86 is already newer than 82 2024-04-26T01:24:58.812 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 86 (ObjNum 118 snap 25 seq_num 118) dirty exists 2024-04-26T01:24:58.812 INFO:tasks.rados.rados.0.smithi181.stdout:311: left oid 50 (ObjNum 118 snap 25 seq_num 118) 2024-04-26T01:24:58.820 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:24:58.824+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:24:58.850 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:58.851 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:58.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:24:58.921 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:58.923+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:58.987 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:58.987 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:59.003 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:24:59.021 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:59.023+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:59.122 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:59.127+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:59.126 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:59.126 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:59.140 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:24:59.196 INFO:tasks.rados.rados.0.smithi181.stdout:313: finishing rollback tid 0 to smithi18128752-46 2024-04-26T01:24:59.199 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 114 (ObjNum 113 snap 24 seq_num 113) dirty exists 2024-04-26T01:24:59.223 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:59.227+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:59.262 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:59.263 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:59.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:24:59.298 INFO:tasks.rados.rados.0.smithi181.stdout:313: finishing rollback tid 1 to smithi18128752-46 2024-04-26T01:24:59.298 INFO:tasks.rados.rados.0.smithi181.stdout:313: finishing rollback tid 2 to smithi18128752-46 2024-04-26T01:24:59.298 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 46 v 48 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-04-26T01:24:59.324 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:59.327+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:59.409 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:59.410 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:59.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:24:59.468 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:59.472+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:59.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:24:59.546 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:24:59.551+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:24:59.586 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:59.586 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:59.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:24:59.643 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:24:59.648+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:24:59.664 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:24:59.664 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:24:59.685 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:24:59.685 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:24:59.706 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:24:59.707 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:24:59.722 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:24:59.722 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:24:59.726 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:24:59.726 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:24:59.727 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:24:59.731+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:24:59.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:24:59.743 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:24:59.743 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:24:59.764 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:24:59.764 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:24:59.780 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.7 to 1 (10000) 2024-04-26T01:24:59.787 INFO:tasks.rados.rados.0.smithi181.stdout:311: done (3 left) 2024-04-26T01:24:59.787 INFO:tasks.rados.rados.0.smithi181.stdout:313: done (2 left) 2024-04-26T01:24:59.787 INFO:tasks.rados.rados.0.smithi181.stdout:314: done (1 left) 2024-04-26T01:24:59.787 INFO:tasks.rados.rados.0.smithi181.stdout:315: done (0 left) 2024-04-26T01:24:59.787 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 33 current snap is 26 2024-04-26T01:24:59.787 INFO:tasks.rados.rados.0.smithi181.stdout:316: seq_num 119 ranges {540672=458752,999424=507904,1507328=507904,2015232=458752,2473984=81920} 2024-04-26T01:24:59.787 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:24:59.788 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:24:59.797 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18128752-33 from 540672 to 999424 tid 1 2024-04-26T01:24:59.801 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T01:24:59.802 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:24:59.802 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18128752-33 from 999424 to 1507328 tid 2 2024-04-26T01:24:59.807 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18128752-33 from 1507328 to 2015232 tid 3 2024-04-26T01:24:59.809 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:24:59.810 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:24:59.812 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18128752-33 from 2015232 to 2473984 tid 4 2024-04-26T01:24:59.813 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18128752-33 from 2473984 to 2555904 tid 5 2024-04-26T01:24:59.813 INFO:tasks.rados.rados.0.smithi181.stdout:317: copy_from oid 18 from oid 13 current snap is 26 2024-04-26T01:24:59.813 INFO:tasks.rados.rados.0.smithi181.stdout:318: snap_create 2024-04-26T01:24:59.829 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:24:59.832+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:24:59.829 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:24:59.829 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:24:59.851 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:24:59.852 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:24:59.873 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:24:59.873 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:24:59.894 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:24:59.894 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:24:59.902 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:24:59.902 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:24:59.902 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:24:59.903 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:24:59.915 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:24:59.915 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:24:59.916 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:24:59.930 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:24:59.935+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:24:59.939 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:24:59.939 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:24:59.989 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:25:00.031 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:00.032+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:00.050 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:00.050 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:00.076 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:00.076 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:00.083 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:00.083 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:00.083 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:25:00.083 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:25:00.084 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:00.101 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:25:00.111 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:00.112 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:25:00.130 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-04-26T01:24:59.837473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83477024,"num_objects":58,"num_object_clones":14,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":1,"num_read":541,"num_read_kb":125590,"num_write":775,"num_write_kb":124420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":29905440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1056,"ondisk_log_size":1056,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":588456,"kb_used_data":151924,"kb_used_omap":21,"kb_used_meta":436330,"kb_avail":1320617304,"statfs":{"total":1352914698240,"available":1352312119296,"internally_reserved":0,"allocated":155570176,"data_stored":189260367,"data_compressed":2691257,"data_compressed_allocated":3698688,"data_compressed_original":8269824,"omap_allocated":22331,"internal_metadata":446802117},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,2,10,8,8,9,3,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15712256,"num_objects":5,"num_object_clones":4,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":25,"num_read_kb":24377,"num_write":62,"num_write_kb":18096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001315"},"pg_stats":[{"pgid":"3.f","version":"34'4","reported_seq":291,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.989889+0000","last_change":"2024-04-26T01:24:57.990598+0000","last_active":"2024-04-26T01:24:57.989889+0000","last_peered":"2024-04-26T01:24:57.989889+0000","last_clean":"2024-04-26T01:24:57.989889+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:24:57.989889+0000","last_undegraded":"2024-04-26T01:24:57.989889+0000","last_fullsized":"2024-04-26T01:24:57.989889+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:01.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013064089999999999,"stat_sum":{"num_bytes":753664,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":2212,"num_write":4,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.e","version":"84'71","reported_seq":288,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.970303+0000","last_change":"2024-04-26T01:24:57.970990+0000","last_active":"2024-04-26T01:24:57.970303+0000","last_peered":"2024-04-26T01:24:57.970303+0000","last_clean":"2024-04-26T01:24:57.970303+0000","last_became_active":"2024-04-26T01:24:16.649834+0000","last_became_peered":"2024-04-26T01:24:16.649834+0000","last_unstale":"2024-04-26T01:24:57.970303+0000","last_undegraded":"2024-04-26T01:24:57.970303+0000","last_fullsized":"2024-04-26T01:24:57.970303+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:00.294383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001579198,"stat_sum":{"num_bytes":6569984,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":12431,"num_write":34,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5128192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,14],"acting":[13,9,1,11,8,6,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.d","version":"84'70","reported_seq":187,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:56.985093+0000","last_change":"2024-04-26T01:24:54.984467+0000","last_active":"2024-04-26T01:24:56.985093+0000","last_peered":"2024-04-26T01:24:56.985093+0000","last_clean":"2024-04-26T01:24:56.985093+0000","last_became_active":"2024-04-26T01:24:36.105677+0000","last_became_peered":"2024-04-26T01:24:36.105677+0000","last_unstale":"2024-04-26T01:24:56.985093+0000","last_undegraded":"2024-04-26T01:24:56.985093+0000","last_fullsized":"2024-04-26T01:24:56.985093+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:03.557570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011254609999999999,"stat_sum":{"num_bytes":4849664,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":35,"num_read_kb":9898,"num_write":28,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"3.c","version":"86'86","reported_seq":363,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.312065+0000","last_change":"2024-04-26T01:24:57.996269+0000","last_active":"2024-04-26T01:24:58.312065+0000","last_peered":"2024-04-26T01:24:58.312065+0000","last_clean":"2024-04-26T01:24:58.312065+0000","last_became_active":"2024-04-26T01:24:05.439203+0000","last_became_peered":"2024-04-26T01:24:05.439203+0000","last_unstale":"2024-04-26T01:24:58.312065+0000","last_undegraded":"2024-04-26T01:24:58.312065+0000","last_fullsized":"2024-04-26T01:24:58.312065+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.340360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069984089999999997,"stat_sum":{"num_bytes":7192576,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":3850,"num_write":72,"num_write_kb":11312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,14,5],"acting":[11,8,0,10,6,14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"72'28","reported_seq":329,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.967923+0000","last_change":"2024-04-26T01:24:57.968691+0000","last_active":"2024-04-26T01:24:57.967923+0000","last_peered":"2024-04-26T01:24:57.967923+0000","last_clean":"2024-04-26T01:24:57.967923+0000","last_became_active":"2024-04-26T01:24:36.003321+0000","last_became_peered":"2024-04-26T01:24:36.003321+0000","last_unstale":"2024-04-26T01:24:57.967923+0000","last_undegraded":"2024-04-26T01:24:57.967923+0000","last_fullsized":"2024-04-26T01:24:57.967923+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.109742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077230300000000001,"stat_sum":{"num_bytes":1458176,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":573,"num_write":30,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1949696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,9,6,7,13],"acting":[4,0,1,9,6,7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.a","version":"82'39","reported_seq":319,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.184260+0000","last_change":"2024-04-26T01:24:55.010824+0000","last_active":"2024-04-26T01:24:57.184260+0000","last_peered":"2024-04-26T01:24:57.184260+0000","last_clean":"2024-04-26T01:24:57.184260+0000","last_became_active":"2024-04-26T01:24:36.102041+0000","last_became_peered":"2024-04-26T01:24:36.102041+0000","last_unstale":"2024-04-26T01:24:57.184260+0000","last_undegraded":"2024-04-26T01:24:57.184260+0000","last_fullsized":"2024-04-26T01:24:57.184260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:13.609035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010641629999999999,"stat_sum":{"num_bytes":3293184,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":9684,"num_write":33,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"3.9","version":"82'63","reported_seq":334,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.072104+0000","last_change":"2024-04-26T01:24:58.172448+0000","last_active":"2024-04-26T01:24:58.072104+0000","last_peered":"2024-04-26T01:24:58.072104+0000","last_clean":"2024-04-26T01:24:58.072104+0000","last_became_active":"2024-04-26T01:24:01.291346+0000","last_became_peered":"2024-04-26T01:24:01.291346+0000","last_unstale":"2024-04-26T01:24:58.072104+0000","last_undegraded":"2024-04-26T01:24:58.072104+0000","last_fullsized":"2024-04-26T01:24:58.072104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.850684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100875927,"stat_sum":{"num_bytes":2703360,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":1015,"num_write":23,"num_write_kb":4720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.8","version":"84'113","reported_seq":419,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.047749+0000","last_change":"2024-04-26T01:24:54.796459+0000","last_active":"2024-04-26T01:24:57.047749+0000","last_peered":"2024-04-26T01:24:57.047749+0000","last_clean":"2024-04-26T01:24:57.047749+0000","last_became_active":"2024-04-26T01:24:16.447854+0000","last_became_peered":"2024-04-26T01:24:16.447854+0000","last_unstale":"2024-04-26T01:24:57.047749+0000","last_undegraded":"2024-04-26T01:24:57.047749+0000","last_fullsized":"2024-04-26T01:24:57.047749+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:26.721419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291122,"stat_sum":{"num_bytes":13271040,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":20737,"num_write":72,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4161536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1,9,14,15,6],"acting":[5,0,1,9,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"1.0","version":"34'157","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.968094+0000","last_change":"2024-04-26T01:24:56.353548+0000","last_active":"2024-04-26T01:24:57.968094+0000","last_peered":"2024-04-26T01:24:57.968094+0000","last_clean":"2024-04-26T01:24:57.968094+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:24:57.968094+0000","last_undegraded":"2024-04-26T01:24:57.968094+0000","last_fullsized":"2024-04-26T01:24:57.968094+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:10.087788+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:56.949093+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:24:56.949093+0000","last_peered":"2024-04-26T01:24:56.949093+0000","last_clean":"2024-04-26T01:24:56.949093+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:24:56.949093+0000","last_undegraded":"2024-04-26T01:24:56.949093+0000","last_fullsized":"2024-04-26T01:24:56.949093+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"82'54","reported_seq":348,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.969384+0000","last_change":"2024-04-26T01:24:57.970126+0000","last_active":"2024-04-26T01:24:57.969384+0000","last_peered":"2024-04-26T01:24:57.969384+0000","last_clean":"2024-04-26T01:24:57.969384+0000","last_became_active":"2024-04-26T01:24:16.348216+0000","last_became_peered":"2024-04-26T01:24:16.348216+0000","last_unstale":"2024-04-26T01:24:57.969384+0000","last_undegraded":"2024-04-26T01:24:57.969384+0000","last_fullsized":"2024-04-26T01:24:57.969384+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:13.911417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013350390000000001,"stat_sum":{"num_bytes":6242304,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":1209,"num_write":25,"num_write_kb":6816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.084013+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:24:57.084013+0000","last_peered":"2024-04-26T01:24:57.084013+0000","last_clean":"2024-04-26T01:24:57.084013+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:24:57.084013+0000","last_undegraded":"2024-04-26T01:24:57.084013+0000","last_fullsized":"2024-04-26T01:24:57.084013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"84'41","reported_seq":339,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.967850+0000","last_change":"2024-04-26T01:24:57.968318+0000","last_active":"2024-04-26T01:24:57.967850+0000","last_peered":"2024-04-26T01:24:57.967850+0000","last_clean":"2024-04-26T01:24:57.967850+0000","last_became_active":"2024-04-26T01:24:16.442216+0000","last_became_peered":"2024-04-26T01:24:16.442216+0000","last_unstale":"2024-04-26T01:24:57.967850+0000","last_undegraded":"2024-04-26T01:24:57.967850+0000","last_fullsized":"2024-04-26T01:24:57.967850+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.974461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077858400000000003,"stat_sum":{"num_bytes":4800512,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":11803,"num_write":25,"num_write_kb":9344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,14,13],"acting":[4,11,9,5,10,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.968429+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:24:57.968429+0000","last_peered":"2024-04-26T01:24:57.968429+0000","last_clean":"2024-04-26T01:24:57.968429+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:24:57.968429+0000","last_undegraded":"2024-04-26T01:24:57.968429+0000","last_fullsized":"2024-04-26T01:24:57.968429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"84'134","reported_seq":442,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.971091+0000","last_change":"2024-04-26T01:24:58.171625+0000","last_active":"2024-04-26T01:24:57.971091+0000","last_peered":"2024-04-26T01:24:57.971091+0000","last_clean":"2024-04-26T01:24:57.971091+0000","last_became_active":"2024-04-26T01:24:16.549243+0000","last_became_peered":"2024-04-26T01:24:16.549243+0000","last_unstale":"2024-04-26T01:24:57.971091+0000","last_undegraded":"2024-04-26T01:24:57.971091+0000","last_fullsized":"2024-04-26T01:24:57.971091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:00.754066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20078206200000001,"stat_sum":{"num_bytes":6356992,"num_objects":9,"num_object_clones":2,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":23877,"num_write":94,"num_write_kb":12208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":132,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.084270+0000","last_change":"2024-04-26T01:24:05.176463+0000","last_active":"2024-04-26T01:24:57.084270+0000","last_peered":"2024-04-26T01:24:57.084270+0000","last_clean":"2024-04-26T01:24:57.084270+0000","last_became_active":"2024-04-26T01:24:05.175683+0000","last_became_peered":"2024-04-26T01:24:05.175683+0000","last_unstale":"2024-04-26T01:24:57.084270+0000","last_undegraded":"2024-04-26T01:24:57.084270+0000","last_fullsized":"2024-04-26T01:24:57.084270+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"71'11","reported_seq":316,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.969569+0000","last_change":"2024-04-26T01:24:58.070548+0000","last_active":"2024-04-26T01:24:57.969569+0000","last_peered":"2024-04-26T01:24:57.969569+0000","last_clean":"2024-04-26T01:24:57.969569+0000","last_became_active":"2024-04-26T01:24:36.104415+0000","last_became_peered":"2024-04-26T01:24:36.104415+0000","last_unstale":"2024-04-26T01:24:57.969569+0000","last_undegraded":"2024-04-26T01:24:57.969569+0000","last_fullsized":"2024-04-26T01:24:57.969569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.974237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10157601300000001,"stat_sum":{"num_bytes":901120,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":4532,"num_write":23,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,6,5,3],"acting":[13,7,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":285,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.051922+0000","last_change":"2024-04-26T01:24:54.885623+0000","last_active":"2024-04-26T01:24:57.051922+0000","last_peered":"2024-04-26T01:24:57.051922+0000","last_clean":"2024-04-26T01:24:57.051922+0000","last_became_active":"2024-04-26T01:24:36.199837+0000","last_became_peered":"2024-04-26T01:24:36.199837+0000","last_unstale":"2024-04-26T01:24:57.051922+0000","last_undegraded":"2024-04-26T01:24:57.051922+0000","last_fullsized":"2024-04-26T01:24:57.051922+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:03.011358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011138680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"2.4","version":"0'0","reported_seq":128,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.967523+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:24:57.967523+0000","last_peered":"2024-04-26T01:24:57.967523+0000","last_clean":"2024-04-26T01:24:57.967523+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:24:57.967523+0000","last_undegraded":"2024-04-26T01:24:57.967523+0000","last_fullsized":"2024-04-26T01:24:57.967523+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"84'49","reported_seq":362,"reported_epoch":87,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T01:24:58.899593+0000","last_change":"2024-04-26T01:24:58.900074+0000","last_active":"2024-04-26T01:24:58.899593+0000","last_peered":"2024-04-26T01:24:58.899593+0000","last_clean":"2024-04-26T01:24:58.899593+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:24:58.899593+0000","last_undegraded":"2024-04-26T01:24:58.899593+0000","last_fullsized":"2024-04-26T01:24:58.899593+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:01.017986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101147268,"stat_sum":{"num_bytes":12484608,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":15374,"num_write":29,"num_write_kb":12672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":126,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:56.984629+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:24:56.984629+0000","last_peered":"2024-04-26T01:24:56.984629+0000","last_clean":"2024-04-26T01:24:56.984629+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:24:56.984629+0000","last_undegraded":"2024-04-26T01:24:56.984629+0000","last_fullsized":"2024-04-26T01:24:56.984629+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"86'99","reported_seq":384,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.172763+0000","last_change":"2024-04-26T01:24:58.173669+0000","last_active":"2024-04-26T01:24:58.172763+0000","last_peered":"2024-04-26T01:24:58.172763+0000","last_clean":"2024-04-26T01:24:58.172763+0000","last_became_active":"2024-04-26T01:24:01.091412+0000","last_became_peered":"2024-04-26T01:24:01.091412+0000","last_unstale":"2024-04-26T01:24:58.172763+0000","last_undegraded":"2024-04-26T01:24:58.172763+0000","last_fullsized":"2024-04-26T01:24:58.172763+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:01.180565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.205858445,"stat_sum":{"num_bytes":11894784,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":5632,"num_write":62,"num_write_kb":13216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":130,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.898773+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:24:58.898773+0000","last_peered":"2024-04-26T01:24:58.898773+0000","last_clean":"2024-04-26T01:24:58.898773+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:24:58.898773+0000","last_undegraded":"2024-04-26T01:24:58.898773+0000","last_fullsized":"2024-04-26T01:24:58.898773+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:10.471688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"72'37","reported_seq":345,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.994415+0000","last_change":"2024-04-26T01:24:57.994756+0000","last_active":"2024-04-26T01:24:57.994415+0000","last_peered":"2024-04-26T01:24:57.994415+0000","last_clean":"2024-04-26T01:24:57.994415+0000","last_became_active":"2024-04-26T01:24:36.202766+0000","last_became_peered":"2024-04-26T01:24:36.202766+0000","last_unstale":"2024-04-26T01:24:57.994415+0000","last_undegraded":"2024-04-26T01:24:57.994415+0000","last_fullsized":"2024-04-26T01:24:57.994415+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053620800000000004,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2645,"num_write":18,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,11,7,9],"acting":[0,13,1,15,11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.051573+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:24:57.051573+0000","last_peered":"2024-04-26T01:24:57.051573+0000","last_clean":"2024-04-26T01:24:57.051573+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:24:57.051573+0000","last_undegraded":"2024-04-26T01:24:57.051573+0000","last_fullsized":"2024-04-26T01:24:57.051573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":82886656,"num_objects":56,"num_object_clones":14,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":1,"num_read":405,"num_read_kb":125472,"num_write":572,"num_write_kb":120960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":29376512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147615744,"data_stored":185163776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":899,"ondisk_log_size":899,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":0,"seq":94489280530,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43496,"kb_used_data":12316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328344,"statfs":{"total":96636764160,"available":96592224256,"internally_reserved":0,"allocated":12611584,"data_stored":16120835,"data_compressed":193289,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,0,6,7,2,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45004,"kb_used_data":13824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326836,"statfs":{"total":96636764160,"available":96590680064,"internally_reserved":0,"allocated":14155776,"data_stored":18205337,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313232,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45132,"kb_used_data":13952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326708,"statfs":{"total":96636764160,"available":96590548992,"internally_reserved":0,"allocated":14286848,"data_stored":18848409,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313233,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46848,"kb_used_data":15668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324992,"statfs":{"total":96636764160,"available":96588791808,"internally_reserved":0,"allocated":16044032,"data_stored":20855449,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39356,"kb_used_data":8176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332484,"statfs":{"total":96636764160,"available":96596463616,"internally_reserved":0,"allocated":8372224,"data_stored":9521817,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":5380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":5509120,"data_stored":5727344,"data_compressed":218307,"data_compressed_allocated":294912,"data_compressed_original":1032192,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313232,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42568,"kb_used_data":11388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329272,"statfs":{"total":96636764160,"available":96593174528,"internally_reserved":0,"allocated":11661312,"data_stored":15178393,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313233,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45900,"kb_used_data":10560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325940,"statfs":{"total":96636764160,"available":96589762560,"internally_reserved":0,"allocated":10813440,"data_stored":12345936,"data_compressed":184252,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,5,0,1,6,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313233,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45604,"kb_used_data":14424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326236,"statfs":{"total":96636764160,"available":96590065664,"internally_reserved":0,"allocated":14770176,"data_stored":16140953,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30560,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341280,"statfs":{"total":96636764160,"available":96605470720,"internally_reserved":0,"allocated":3616768,"data_stored":3355216,"data_compressed":184252,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45288,"kb_used_data":14108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326552,"statfs":{"total":96636764160,"available":96590389248,"internally_reserved":0,"allocated":14446592,"data_stored":18950809,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313232,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33964,"kb_used_data":6944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337876,"statfs":{"total":96636764160,"available":96601985024,"internally_reserved":0,"allocated":7110656,"data_stored":7096985,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313234,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43676,"kb_used_data":12492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328164,"statfs":{"total":96636764160,"available":96592039936,"internally_reserved":0,"allocated":12791808,"data_stored":13669162,"data_compressed":190262,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313233,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44500,"kb_used_data":9160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327340,"statfs":{"total":96636764160,"available":96591196160,"internally_reserved":0,"allocated":9379840,"data_stored":13243722,"data_compressed":222879,"data_compressed_allocated":303104,"data_compressed_original":1036288,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13291520,"data_stored":17952768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13312000,"data_stored":18137088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4497408,"data_stored":4497408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10686464,"data_stored":14467072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10604544,"data_stored":12283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13795328,"data_stored":15413248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2654208,"data_stored":2654208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11612160,"data_stored":15388672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13471744,"data_stored":18223104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6135808,"data_stored":6385664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11804672,"data_stored":12947456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8331264,"data_stored":11993088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":15568896,"data_stored":21008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":7397376,"data_stored":8810496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:25:00.132 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:25:00.134 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing copy_from racing read to smithi18128752-18 2024-04-26T01:25:00.134 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:00.135+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:00.235 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:00.235+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:00.255 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:00.255 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:25:00.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:25:00.255 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:00.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.255 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.as1 66/63 ECSubWrite(tid=26, reqid=osd.3.0:25, at_version=0'0, trim_to=0'0, roll_forward_to=66'21))", 2024-04-26T01:25:00.255 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.453722+0000", 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.803855411000001, 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201058036, 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453722+0000", 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453722+0000", 2024-04-26T01:25:00.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453724+0000", 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1100000000000001e-06 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453753+0000", 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8382999999999999e-05 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453805+0000", 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2707999999999997e-05 2024-04-26T01:25:00.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.453830+0000", 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4873999999999999e-05 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.554095+0000", 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10026523900000001 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.554111+0000", 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6025999999999999e-05 2024-04-26T01:25:00.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.654721+0000", 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10060960100000001 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.654780+0000", 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9095000000000003e-05 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:542 3.ds0 3:b6ed1d05:::smithi18128752-41:head [copy-from ver 70] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.418242083999999, 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99677863200000005, 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 542 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839338+0000", 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7669999999999999e-06 2024-04-26T01:25:00.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839350+0000", 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2109e-05 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839372+0000", 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.162e-05 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839396+0000", 2024-04-26T01:25:00.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3898999999999999e-05 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986378+0000", 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14698248899999999 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986395+0000", 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6545000000000002e-05 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.196668+0000", 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.210273194 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.196889+0000", 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022093199999999999 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.626756+0000", 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.42986713199999999 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.265 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.635172+0000", 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084160469999999994 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.835968+0000", 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20079548799999999 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.836114+0000", 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014641 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.266 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:543 3.ds0 3:b6ed1d05:::smithi18128752-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.418142375999999, 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.79564045900000002, 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 543 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.267 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839437+0000", 2024-04-26T01:25:00.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6130000000000001e-06 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839444+0000", 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7700000000000004e-06 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839465+0000", 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0970000000000001e-05 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839486+0000", 2024-04-26T01:25:00.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1199000000000002e-05 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986531+0000", 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14704526000000001 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986557+0000", 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5701999999999999e-05 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.270 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197379+0000", 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21082251799999999 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197562+0000", 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000182652 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.634726+0000", 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43716451899999997 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.634892+0000", 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016576000000000001 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.635076+0000", 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018349600000000001 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 72/63 ECSubWriteReply(tid=26, last_complete=71'56, committed=1, applied=1))", 2024-04-26T01:25:00.272 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:43.920731+0000", 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.336846361999999, 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20149829699999999, 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.920731+0000", 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.920731+0000", 2024-04-26T01:25:00.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.920733+0000", 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4300000000000001e-06 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.920742+0000", 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0399999999999998e-06 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.920764+0000", 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2549999999999999e-05 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020834+0000", 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100069492 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.122170+0000", 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101336602 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.122230+0000", 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9182999999999999e-05 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 72/63 ECSubWriteReply(tid=26, last_complete=71'56, committed=1, applied=1))", 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:43.921086+0000", 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.336491908999999, 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20149175, 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.276 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.921086+0000", 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.921086+0000", 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.921087+0000", 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.767e-06 2024-04-26T01:25:00.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.921096+0000", 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9120000000000004e-06 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:43.921124+0000", 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7863000000000001e-05 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.021207+0000", 2024-04-26T01:25:00.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100083118 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.122541+0000", 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101333747 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.122577+0000", 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6343000000000001e-05 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 72/63 ECSubReadReply(tid=27, attrs_read=0))", 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.020968+0000", 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.236610034999998, 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202231046, 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020968+0000", 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020968+0000", 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020968+0000", 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4499999999999999e-07 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020978+0000", 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0244999999999999e-05 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.020983+0000", 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.383e-06 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.121054+0000", 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100070198 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.222889+0000", 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10183526900000001 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.223199+0000", 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030980599999999998 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 72/63 ECSubWriteReply(tid=31, last_complete=72'60, committed=1, applied=1))", 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.429619+0000", 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.827958205, 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200345579, 2024-04-26T01:25:00.283 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429619+0000", 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429619+0000", 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429620+0000", 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4399999999999998e-07 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429626+0000", 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3310000000000003e-06 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429638+0000", 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2058999999999999e-05 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429659+0000", 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0829e-05 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.629920+0000", 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20026171700000001 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.629965+0000", 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4699e-05 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.286 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 72/63 ECSubWriteReply(tid=30, last_complete=72'58, committed=1, applied=1))", 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.429771+0000", 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.827806529, 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20039254200000001, 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429771+0000", 2024-04-26T01:25:00.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429771+0000", 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429772+0000", 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0800000000000005e-07 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429774+0000", 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8959999999999999e-06 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429783+0000", 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2640000000000008e-06 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429789+0000", 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.336e-06 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.630125+0000", 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20033601500000001 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.630164+0000", 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8522999999999997e-05 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 72/63 ECSubWriteReply(tid=31, last_complete=72'60, committed=1, applied=1))", 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.429957+0000", 2024-04-26T01:25:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.827620121000001, 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20041144999999999, 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429957+0000", 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429957+0000", 2024-04-26T01:25:00.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429958+0000", 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6100000000000002e-07 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429961+0000", 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0570000000000001e-06 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429969+0000", 2024-04-26T01:25:00.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4039999999999995e-06 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.429975+0000", 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8930000000000002e-06 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.630339+0000", 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20036380600000001 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.630369+0000", 2024-04-26T01:25:00.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9929000000000001e-05 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.14.21:4 3.ds0 3:b136579b:::smithi18128752-36:head [assert-version v14,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.925621+0000", 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.331956535, 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81766490700000005, 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925621+0000", 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925621+0000", 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925624+0000", 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7159999999999999e-06 2024-04-26T01:25:00.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925638+0000", 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.447e-05 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925677+0000", 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9135999999999999e-05 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925692+0000", 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4902999999999999e-05 2024-04-26T01:25:00.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925796+0000", 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103651 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925804+0000", 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4950000000000008e-06 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.133516+0000", 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20771204700000001 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.133724+0000", 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020719900000000001 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743286+0000", 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60956228999999995 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:693 3.ds0 3:b6ed1d05:::smithi18128752-41:head [append 0~0] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.988574+0000", 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.26900328, 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55445387800000001, 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.299 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.303 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 693 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988574+0000", 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988574+0000", 2024-04-26T01:25:00.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988578+0000", 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8530000000000002e-06 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988594+0000", 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.624e-05 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988631+0000", 2024-04-26T01:25:00.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6519000000000001e-05 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988651+0000", 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0233000000000001e-05 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.134541+0000", 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14588947799999999 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.134724+0000", 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018301299999999999 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.135949+0000", 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012256700000000001 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.339321+0000", 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.203372152 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.542894+0000", 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20357222899999999 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543028+0000", 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013449100000000001 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:694 3.ds0 3:b6ed1d05:::smithi18128752-41:head [rollback 13] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.988685+0000", 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.268892644999999, 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55479098800000004, 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 694 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988685+0000", 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988685+0000", 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988686+0000", 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0359999999999999e-06 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988691+0000", 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8799999999999999e-06 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988702+0000", 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1036000000000001e-05 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988711+0000", 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9830000000000002e-06 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.237077+0000", 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24836664 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.237256+0000", 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017854599999999999 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.238335+0000", 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00107851 2024-04-26T01:25:00.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.339397+0000", 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101062768 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543374+0000", 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20397695099999999 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543476+0000", 2024-04-26T01:25:00.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010163800000000001 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:695 3.ds0 3:b6ed1d05:::smithi18128752-41:head [append 0~0] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.988724+0000", 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.268853843, 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.75505351200000004, 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.314 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 695 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988724+0000", 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988724+0000", 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988725+0000", 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5000000000000001e-07 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988740+0000", 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5573000000000002e-05 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988750+0000", 2024-04-26T01:25:00.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0183000000000001e-05 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988758+0000", 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.14e-06 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.239213+0000", 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.250454385 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.239440+0000", 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022750900000000001 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.240667+0000", 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001226783 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.339400+0000", 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098732746999999996 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.318 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743591+0000", 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40419128900000001 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743777+0000", 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018605299999999999 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 78/63 ECSubReadReply(tid=37, attrs_read=0))", 2024-04-26T01:25:00.319 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.137388+0000", 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.120189922, 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.204342247, 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137388+0000", 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137388+0000", 2024-04-26T01:25:00.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137388+0000", 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137388+0000", 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137466+0000", 2024-04-26T01:25:00.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8108999999999994e-05 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137483+0000", 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7286e-05 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341636+0000", 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20415344999999999 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341730+0000", 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3401999999999998e-05 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 78/63 ECSubReadReply(tid=37, attrs_read=0))", 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.137473+0000", 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.120104177, 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20435170999999999, 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.323 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137473+0000", 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137473+0000", 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137475+0000", 2024-04-26T01:25:00.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6500000000000001e-06 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137501+0000", 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6216000000000001e-05 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137518+0000", 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.632e-05 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.137530+0000", 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2357e-05 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341773+0000", 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.204242588 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341825+0000", 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2578999999999999e-05 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.326 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.3s1 78/63 ECSubRead(tid=293, to_read={3:c5d16e9b:::smithi18128752-38:head=0,225280,0}, subchunks={3:c5d16e9b:::smithi18128752-38:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.241396+0000", 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.016181273999999, 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20059785299999999, 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.241396+0000", 2024-04-26T01:25:00.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.241396+0000", 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.241397+0000", 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0999999999999998e-07 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.241404+0000", 2024-04-26T01:25:00.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6560000000000003e-06 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.241417+0000", 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3176999999999999e-05 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341515+0000", 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100098127 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.441732+0000", 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100217468 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.441994+0000", 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00026181499999999999 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 78/63 ECSubReadReply(tid=37, attrs_read=0))", 2024-04-26T01:25:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.335466+0000", 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.9221114640000003, 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40784600500000001, 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335466+0000", 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335466+0000", 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335468+0000", 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.437e-06 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335499+0000", 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1173000000000001e-05 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335534+0000", 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5451999999999999e-05 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.435652+0000", 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10011783 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.742921+0000", 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30726856800000002 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743312+0000", 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039154499999999998 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 78/63 ECSubWriteReply(tid=40, last_complete=78'64, committed=1, applied=1))", 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.341870+0000", 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.9157079909999997, 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20053654100000001, 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.334 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341870+0000", 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341870+0000", 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341870+0000", 2024-04-26T01:25:00.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8000000000000004e-07 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341877+0000", 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3999999999999997e-06 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341896+0000", 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8723e-05 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.341912+0000", 2024-04-26T01:25:00.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6500000000000001e-05 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.542369+0000", 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200457142 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.542406+0000", 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6896e-05 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 78/63 ECSubWriteReply(tid=40, last_complete=78'64, committed=1, applied=1))", 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.342765+0000", 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.9148128389999997, 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20046103300000001, 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.342765+0000", 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.342765+0000", 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.342766+0000", 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.547e-06 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.342773+0000", 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6470000000000001e-06 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.342911+0000", 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138174 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.342944+0000", 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3129000000000001e-05 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543204+0000", 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20025938300000001 2024-04-26T01:25:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543226+0000", 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2152999999999999e-05 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.as1 84/63 ECSubRead(tid=47, to_read={3:56e8b2fd:::smithi18128752-12:head=0,823296,0}, subchunks={3:56e8b2fd:::smithi18128752-12:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:56.073377+0000", 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1842005889999996, 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202156474, 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073377+0000", 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073377+0000", 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073379+0000", 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2869999999999999e-06 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073400+0000", 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0525000000000001e-05 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073441+0000", 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.142e-05 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.173613+0000", 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100172252 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.273909+0000", 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100295708 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.275533+0000", 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016242819999999999 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.345 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.345 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.345 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:00.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:25:00.347 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 1 to smithi18128752-33 2024-04-26T01:25:00.347 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:00.348+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:00.448 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:00.451+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:00.473 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:00.473 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:25:00.491 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-04-26T01:24:59.837473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83477024,"num_objects":58,"num_object_clones":14,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":1,"num_read":541,"num_read_kb":125590,"num_write":775,"num_write_kb":124420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":29905440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1056,"ondisk_log_size":1056,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":588456,"kb_used_data":151924,"kb_used_omap":21,"kb_used_meta":436330,"kb_avail":1320617304,"statfs":{"total":1352914698240,"available":1352312119296,"internally_reserved":0,"allocated":155570176,"data_stored":189260367,"data_compressed":2691257,"data_compressed_allocated":3698688,"data_compressed_original":8269824,"omap_allocated":22331,"internal_metadata":446802117},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,2,10,8,8,9,3,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15712256,"num_objects":5,"num_object_clones":4,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":25,"num_read_kb":24377,"num_write":62,"num_write_kb":18096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001315"},"pg_stats":[{"pgid":"3.f","version":"34'4","reported_seq":291,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.989889+0000","last_change":"2024-04-26T01:24:57.990598+0000","last_active":"2024-04-26T01:24:57.989889+0000","last_peered":"2024-04-26T01:24:57.989889+0000","last_clean":"2024-04-26T01:24:57.989889+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:24:57.989889+0000","last_undegraded":"2024-04-26T01:24:57.989889+0000","last_fullsized":"2024-04-26T01:24:57.989889+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:01.949381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013064089999999999,"stat_sum":{"num_bytes":753664,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":2212,"num_write":4,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.e","version":"84'71","reported_seq":288,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.970303+0000","last_change":"2024-04-26T01:24:57.970990+0000","last_active":"2024-04-26T01:24:57.970303+0000","last_peered":"2024-04-26T01:24:57.970303+0000","last_clean":"2024-04-26T01:24:57.970303+0000","last_became_active":"2024-04-26T01:24:16.649834+0000","last_became_peered":"2024-04-26T01:24:16.649834+0000","last_unstale":"2024-04-26T01:24:57.970303+0000","last_undegraded":"2024-04-26T01:24:57.970303+0000","last_fullsized":"2024-04-26T01:24:57.970303+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:00.294383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001579198,"stat_sum":{"num_bytes":6569984,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":12431,"num_write":34,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5128192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,14],"acting":[13,9,1,11,8,6,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.d","version":"84'70","reported_seq":187,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:56.985093+0000","last_change":"2024-04-26T01:24:54.984467+0000","last_active":"2024-04-26T01:24:56.985093+0000","last_peered":"2024-04-26T01:24:56.985093+0000","last_clean":"2024-04-26T01:24:56.985093+0000","last_became_active":"2024-04-26T01:24:36.105677+0000","last_became_peered":"2024-04-26T01:24:36.105677+0000","last_unstale":"2024-04-26T01:24:56.985093+0000","last_undegraded":"2024-04-26T01:24:56.985093+0000","last_fullsized":"2024-04-26T01:24:56.985093+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:03.557570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011254609999999999,"stat_sum":{"num_bytes":4849664,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":35,"num_read_kb":9898,"num_write":28,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"3.c","version":"86'86","reported_seq":363,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.312065+0000","last_change":"2024-04-26T01:24:57.996269+0000","last_active":"2024-04-26T01:24:58.312065+0000","last_peered":"2024-04-26T01:24:58.312065+0000","last_clean":"2024-04-26T01:24:58.312065+0000","last_became_active":"2024-04-26T01:24:05.439203+0000","last_became_peered":"2024-04-26T01:24:05.439203+0000","last_unstale":"2024-04-26T01:24:58.312065+0000","last_undegraded":"2024-04-26T01:24:58.312065+0000","last_fullsized":"2024-04-26T01:24:58.312065+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.340360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069984089999999997,"stat_sum":{"num_bytes":7192576,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":3850,"num_write":72,"num_write_kb":11312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,14,5],"acting":[11,8,0,10,6,14,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"72'28","reported_seq":329,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.967923+0000","last_change":"2024-04-26T01:24:57.968691+0000","last_active":"2024-04-26T01:24:57.967923+0000","last_peered":"2024-04-26T01:24:57.967923+0000","last_clean":"2024-04-26T01:24:57.967923+0000","last_became_active":"2024-04-26T01:24:36.003321+0000","last_became_peered":"2024-04-26T01:24:36.003321+0000","last_unstale":"2024-04-26T01:24:57.967923+0000","last_undegraded":"2024-04-26T01:24:57.967923+0000","last_fullsized":"2024-04-26T01:24:57.967923+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.109742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077230300000000001,"stat_sum":{"num_bytes":1458176,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":573,"num_write":30,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1949696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,9,6,7,13],"acting":[4,0,1,9,6,7,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.a","version":"82'39","reported_seq":319,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.184260+0000","last_change":"2024-04-26T01:24:55.010824+0000","last_active":"2024-04-26T01:24:57.184260+0000","last_peered":"2024-04-26T01:24:57.184260+0000","last_clean":"2024-04-26T01:24:57.184260+0000","last_became_active":"2024-04-26T01:24:36.102041+0000","last_became_peered":"2024-04-26T01:24:36.102041+0000","last_unstale":"2024-04-26T01:24:57.184260+0000","last_undegraded":"2024-04-26T01:24:57.184260+0000","last_fullsized":"2024-04-26T01:24:57.184260+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:13.609035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010641629999999999,"stat_sum":{"num_bytes":3293184,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":9684,"num_write":33,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"3.9","version":"82'63","reported_seq":334,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.072104+0000","last_change":"2024-04-26T01:24:58.172448+0000","last_active":"2024-04-26T01:24:58.072104+0000","last_peered":"2024-04-26T01:24:58.072104+0000","last_clean":"2024-04-26T01:24:58.072104+0000","last_became_active":"2024-04-26T01:24:01.291346+0000","last_became_peered":"2024-04-26T01:24:01.291346+0000","last_unstale":"2024-04-26T01:24:58.072104+0000","last_undegraded":"2024-04-26T01:24:58.072104+0000","last_fullsized":"2024-04-26T01:24:58.072104+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.850684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100875927,"stat_sum":{"num_bytes":2703360,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":1015,"num_write":23,"num_write_kb":4720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.8","version":"84'113","reported_seq":419,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.047749+0000","last_change":"2024-04-26T01:24:54.796459+0000","last_active":"2024-04-26T01:24:57.047749+0000","last_peered":"2024-04-26T01:24:57.047749+0000","last_clean":"2024-04-26T01:24:57.047749+0000","last_became_active":"2024-04-26T01:24:16.447854+0000","last_became_peered":"2024-04-26T01:24:16.447854+0000","last_unstale":"2024-04-26T01:24:57.047749+0000","last_undegraded":"2024-04-26T01:24:57.047749+0000","last_fullsized":"2024-04-26T01:24:57.047749+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:26.721419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001291122,"stat_sum":{"num_bytes":13271040,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":20737,"num_write":72,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4161536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1,9,14,15,6],"acting":[5,0,1,9,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"1.0","version":"34'157","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.968094+0000","last_change":"2024-04-26T01:24:56.353548+0000","last_active":"2024-04-26T01:24:57.968094+0000","last_peered":"2024-04-26T01:24:57.968094+0000","last_clean":"2024-04-26T01:24:57.968094+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:24:57.968094+0000","last_undegraded":"2024-04-26T01:24:57.968094+0000","last_fullsized":"2024-04-26T01:24:57.968094+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:10.087788+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:56.949093+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:24:56.949093+0000","last_peered":"2024-04-26T01:24:56.949093+0000","last_clean":"2024-04-26T01:24:56.949093+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:24:56.949093+0000","last_undegraded":"2024-04-26T01:24:56.949093+0000","last_fullsized":"2024-04-26T01:24:56.949093+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"82'54","reported_seq":348,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.969384+0000","last_change":"2024-04-26T01:24:57.970126+0000","last_active":"2024-04-26T01:24:57.969384+0000","last_peered":"2024-04-26T01:24:57.969384+0000","last_clean":"2024-04-26T01:24:57.969384+0000","last_became_active":"2024-04-26T01:24:16.348216+0000","last_became_peered":"2024-04-26T01:24:16.348216+0000","last_unstale":"2024-04-26T01:24:57.969384+0000","last_undegraded":"2024-04-26T01:24:57.969384+0000","last_fullsized":"2024-04-26T01:24:57.969384+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:13.911417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013350390000000001,"stat_sum":{"num_bytes":6242304,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":1209,"num_write":25,"num_write_kb":6816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.084013+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:24:57.084013+0000","last_peered":"2024-04-26T01:24:57.084013+0000","last_clean":"2024-04-26T01:24:57.084013+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:24:57.084013+0000","last_undegraded":"2024-04-26T01:24:57.084013+0000","last_fullsized":"2024-04-26T01:24:57.084013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"84'41","reported_seq":339,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.967850+0000","last_change":"2024-04-26T01:24:57.968318+0000","last_active":"2024-04-26T01:24:57.967850+0000","last_peered":"2024-04-26T01:24:57.967850+0000","last_clean":"2024-04-26T01:24:57.967850+0000","last_became_active":"2024-04-26T01:24:16.442216+0000","last_became_peered":"2024-04-26T01:24:16.442216+0000","last_unstale":"2024-04-26T01:24:57.967850+0000","last_undegraded":"2024-04-26T01:24:57.967850+0000","last_fullsized":"2024-04-26T01:24:57.967850+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.974461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077858400000000003,"stat_sum":{"num_bytes":4800512,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":11803,"num_write":25,"num_write_kb":9344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,14,13],"acting":[4,11,9,5,10,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.968429+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:24:57.968429+0000","last_peered":"2024-04-26T01:24:57.968429+0000","last_clean":"2024-04-26T01:24:57.968429+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:24:57.968429+0000","last_undegraded":"2024-04-26T01:24:57.968429+0000","last_fullsized":"2024-04-26T01:24:57.968429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"84'134","reported_seq":442,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.971091+0000","last_change":"2024-04-26T01:24:58.171625+0000","last_active":"2024-04-26T01:24:57.971091+0000","last_peered":"2024-04-26T01:24:57.971091+0000","last_clean":"2024-04-26T01:24:57.971091+0000","last_became_active":"2024-04-26T01:24:16.549243+0000","last_became_peered":"2024-04-26T01:24:16.549243+0000","last_unstale":"2024-04-26T01:24:57.971091+0000","last_undegraded":"2024-04-26T01:24:57.971091+0000","last_fullsized":"2024-04-26T01:24:57.971091+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:00.754066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20078206200000001,"stat_sum":{"num_bytes":6356992,"num_objects":9,"num_object_clones":2,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":23877,"num_write":94,"num_write_kb":12208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":132,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.084270+0000","last_change":"2024-04-26T01:24:05.176463+0000","last_active":"2024-04-26T01:24:57.084270+0000","last_peered":"2024-04-26T01:24:57.084270+0000","last_clean":"2024-04-26T01:24:57.084270+0000","last_became_active":"2024-04-26T01:24:05.175683+0000","last_became_peered":"2024-04-26T01:24:05.175683+0000","last_unstale":"2024-04-26T01:24:57.084270+0000","last_undegraded":"2024-04-26T01:24:57.084270+0000","last_fullsized":"2024-04-26T01:24:57.084270+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"71'11","reported_seq":316,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.969569+0000","last_change":"2024-04-26T01:24:58.070548+0000","last_active":"2024-04-26T01:24:57.969569+0000","last_peered":"2024-04-26T01:24:57.969569+0000","last_clean":"2024-04-26T01:24:57.969569+0000","last_became_active":"2024-04-26T01:24:36.104415+0000","last_became_peered":"2024-04-26T01:24:36.104415+0000","last_unstale":"2024-04-26T01:24:57.969569+0000","last_undegraded":"2024-04-26T01:24:57.969569+0000","last_fullsized":"2024-04-26T01:24:57.969569+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.974237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10157601300000001,"stat_sum":{"num_bytes":901120,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":4532,"num_write":23,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,6,5,3],"acting":[13,7,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":285,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.051922+0000","last_change":"2024-04-26T01:24:54.885623+0000","last_active":"2024-04-26T01:24:57.051922+0000","last_peered":"2024-04-26T01:24:57.051922+0000","last_clean":"2024-04-26T01:24:57.051922+0000","last_became_active":"2024-04-26T01:24:36.199837+0000","last_became_peered":"2024-04-26T01:24:36.199837+0000","last_unstale":"2024-04-26T01:24:57.051922+0000","last_undegraded":"2024-04-26T01:24:57.051922+0000","last_fullsized":"2024-04-26T01:24:57.051922+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:03.011358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011138680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"13","length":"5"}]},{"pgid":"2.4","version":"0'0","reported_seq":128,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.967523+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:24:57.967523+0000","last_peered":"2024-04-26T01:24:57.967523+0000","last_clean":"2024-04-26T01:24:57.967523+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:24:57.967523+0000","last_undegraded":"2024-04-26T01:24:57.967523+0000","last_fullsized":"2024-04-26T01:24:57.967523+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"84'49","reported_seq":362,"reported_epoch":87,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T01:24:58.899593+0000","last_change":"2024-04-26T01:24:58.900074+0000","last_active":"2024-04-26T01:24:58.899593+0000","last_peered":"2024-04-26T01:24:58.899593+0000","last_clean":"2024-04-26T01:24:58.899593+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:24:58.899593+0000","last_undegraded":"2024-04-26T01:24:58.899593+0000","last_fullsized":"2024-04-26T01:24:58.899593+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:01.017986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101147268,"stat_sum":{"num_bytes":12484608,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":15374,"num_write":29,"num_write_kb":12672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":126,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:56.984629+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:24:56.984629+0000","last_peered":"2024-04-26T01:24:56.984629+0000","last_clean":"2024-04-26T01:24:56.984629+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:24:56.984629+0000","last_undegraded":"2024-04-26T01:24:56.984629+0000","last_fullsized":"2024-04-26T01:24:56.984629+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"86'99","reported_seq":384,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.172763+0000","last_change":"2024-04-26T01:24:58.173669+0000","last_active":"2024-04-26T01:24:58.172763+0000","last_peered":"2024-04-26T01:24:58.172763+0000","last_clean":"2024-04-26T01:24:58.172763+0000","last_became_active":"2024-04-26T01:24:01.091412+0000","last_became_peered":"2024-04-26T01:24:01.091412+0000","last_unstale":"2024-04-26T01:24:58.172763+0000","last_undegraded":"2024-04-26T01:24:58.172763+0000","last_fullsized":"2024-04-26T01:24:58.172763+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:01.180565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.205858445,"stat_sum":{"num_bytes":11894784,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":5632,"num_write":62,"num_write_kb":13216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":130,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T01:24:58.898773+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:24:58.898773+0000","last_peered":"2024-04-26T01:24:58.898773+0000","last_clean":"2024-04-26T01:24:58.898773+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:24:58.898773+0000","last_undegraded":"2024-04-26T01:24:58.898773+0000","last_fullsized":"2024-04-26T01:24:58.898773+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:10.471688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"72'37","reported_seq":345,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.994415+0000","last_change":"2024-04-26T01:24:57.994756+0000","last_active":"2024-04-26T01:24:57.994415+0000","last_peered":"2024-04-26T01:24:57.994415+0000","last_clean":"2024-04-26T01:24:57.994415+0000","last_became_active":"2024-04-26T01:24:36.202766+0000","last_became_peered":"2024-04-26T01:24:36.202766+0000","last_unstale":"2024-04-26T01:24:57.994415+0000","last_undegraded":"2024-04-26T01:24:57.994415+0000","last_fullsized":"2024-04-26T01:24:57.994415+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.530969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053620800000000004,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2645,"num_write":18,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,11,7,9],"acting":[0,13,1,15,11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"5"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":127,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T01:24:57.051573+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:24:57.051573+0000","last_peered":"2024-04-26T01:24:57.051573+0000","last_clean":"2024-04-26T01:24:57.051573+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:24:57.051573+0000","last_undegraded":"2024-04-26T01:24:57.051573+0000","last_fullsized":"2024-04-26T01:24:57.051573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":82886656,"num_objects":56,"num_object_clones":14,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":1,"num_read":405,"num_read_kb":125472,"num_write":572,"num_write_kb":120960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":29376512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147615744,"data_stored":185163776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":899,"ondisk_log_size":899,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":0,"seq":94489280530,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43496,"kb_used_data":12316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328344,"statfs":{"total":96636764160,"available":96592224256,"internally_reserved":0,"allocated":12611584,"data_stored":16120835,"data_compressed":193289,"data_compressed_allocated":266240,"data_compressed_original":532480,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,0,6,7,2,0,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45004,"kb_used_data":13824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326836,"statfs":{"total":96636764160,"available":96590680064,"internally_reserved":0,"allocated":14155776,"data_stored":18205337,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313232,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45132,"kb_used_data":13952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326708,"statfs":{"total":96636764160,"available":96590548992,"internally_reserved":0,"allocated":14286848,"data_stored":18848409,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313233,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46848,"kb_used_data":15668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324992,"statfs":{"total":96636764160,"available":96588791808,"internally_reserved":0,"allocated":16044032,"data_stored":20855449,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39356,"kb_used_data":8176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332484,"statfs":{"total":96636764160,"available":96596463616,"internally_reserved":0,"allocated":8372224,"data_stored":9521817,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":5380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":5509120,"data_stored":5727344,"data_compressed":218307,"data_compressed_allocated":294912,"data_compressed_original":1032192,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313232,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42568,"kb_used_data":11388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329272,"statfs":{"total":96636764160,"available":96593174528,"internally_reserved":0,"allocated":11661312,"data_stored":15178393,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313233,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45900,"kb_used_data":10560,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325940,"statfs":{"total":96636764160,"available":96589762560,"internally_reserved":0,"allocated":10813440,"data_stored":12345936,"data_compressed":184252,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,5,0,1,6,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313233,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45604,"kb_used_data":14424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326236,"statfs":{"total":96636764160,"available":96590065664,"internally_reserved":0,"allocated":14770176,"data_stored":16140953,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30560,"kb_used_data":3532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341280,"statfs":{"total":96636764160,"available":96605470720,"internally_reserved":0,"allocated":3616768,"data_stored":3355216,"data_compressed":184252,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313233,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45288,"kb_used_data":14108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326552,"statfs":{"total":96636764160,"available":96590389248,"internally_reserved":0,"allocated":14446592,"data_stored":18950809,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313232,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33964,"kb_used_data":6944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337876,"statfs":{"total":96636764160,"available":96601985024,"internally_reserved":0,"allocated":7110656,"data_stored":7096985,"data_compressed":187252,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313234,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43676,"kb_used_data":12492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328164,"statfs":{"total":96636764160,"available":96592039936,"internally_reserved":0,"allocated":12791808,"data_stored":13669162,"data_compressed":190262,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313233,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44500,"kb_used_data":9160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327340,"statfs":{"total":96636764160,"available":96591196160,"internally_reserved":0,"allocated":9379840,"data_stored":13243722,"data_compressed":222879,"data_compressed_allocated":303104,"data_compressed_original":1036288,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13291520,"data_stored":17952768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13312000,"data_stored":18137088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4497408,"data_stored":4497408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10686464,"data_stored":14467072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10604544,"data_stored":12283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13795328,"data_stored":15413248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2654208,"data_stored":2654208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11612160,"data_stored":15388672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13471744,"data_stored":18223104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6135808,"data_stored":6385664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11804672,"data_stored":12947456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":131072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8331264,"data_stored":11993088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":15568896,"data_stored":21008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":7397376,"data_stored":8810496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:25:00.493 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-26T01:25:00.493 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:25:00.493 INFO:tasks.thrashosds.thrasher:Adding osd 12 2024-04-26T01:25:00.494 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 12 2024-04-26T01:25:00.514 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:00.514 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:00.514 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:25:00.514 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:00.517 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 2 to smithi18128752-33 2024-04-26T01:25:00.517 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 3 to smithi18128752-33 2024-04-26T01:25:00.518 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 4 to smithi18128752-33 2024-04-26T01:25:00.518 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing copy_from to smithi18128752-18 2024-04-26T01:25:00.518 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 18 v 108 (ObjNum 91 snap 17 seq_num 91) dirty exists 2024-04-26T01:25:00.529 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:25:00.549 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:00.551+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:00.618 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 5 to smithi18128752-33 2024-04-26T01:25:00.650 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:00.650 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:00.650 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:25:00.650 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:25:00.650 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:00.650 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:00.652+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:00.664 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:25:00.751 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:00.755+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:00.789 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-04-26T01:25:00.794 INFO:tasks.rados.rados.0.smithi181.stdout:317: done (2 left) 2024-04-26T01:25:00.794 INFO:tasks.rados.rados.0.smithi181.stdout:318: done (1 left) 2024-04-26T01:25:00.794 INFO:tasks.rados.rados.0.smithi181.stdout:319: snap_create 2024-04-26T01:25:00.807 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T01:25:00.811+0000 7f533fe62640 -1 mon.a@0(leader).osd e89 definitely_dead 0 2024-04-26T01:25:00.811 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 6 to smithi18128752-33 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:150 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.653360325999998, 2024-04-26T01:25:00.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3959368480000001, 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 150 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179064+0000", 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7269999999999999e-06 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179608+0000", 2024-04-26T01:25:00.832 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00054408799999999997 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179641+0000", 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2873000000000002e-05 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179658+0000", 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6863000000000001e-05 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268330+0000", 2024-04-26T01:25:00.833 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0886726819999999 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268562+0000", 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023123100000000001 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.270316+0000", 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017541189999999999 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.834 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474438+0000", 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20412262 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574871+0000", 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100432788 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574998+0000", 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126857 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.835 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:151 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.646985463999997, 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.494027684, 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 151 2024-04-26T01:25:00.836 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185438+0000", 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2170000000000001e-06 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185895+0000", 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000456291 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185927+0000", 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1989000000000002e-05 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.838 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185942+0000", 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5075999999999999e-05 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272106+0000", 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0861648319999999 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272243+0000", 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013630399999999999 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.839 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.273909+0000", 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016665930000000001 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474512+0000", 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20060244399999999 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679353+0000", 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20484124000000001 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.840 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679464+0000", 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110698 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:152 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.646453557999997, 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4937803999999999, 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.841 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 152 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.842 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185969+0000", 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.167e-06 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185975+0000", 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5029999999999997e-06 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.843 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185986+0000", 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1283e-05 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185994+0000", 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6060000000000003e-06 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372230+0000", 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.186235766 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.844 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372376+0000", 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014574 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.373451+0000", 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010752890000000001 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474514+0000", 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101062945 2024-04-26T01:25:00.845 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679632+0000", 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20511794799999999 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679748+0000", 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116153 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.846 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:153 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.646415181999998, 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.900776316, 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 153 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.847 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186007+0000", 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6700000000000002e-07 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.848 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186011+0000", 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5559999999999999e-06 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186025+0000", 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3873000000000001e-05 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186035+0000", 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0526000000000001e-05 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.849 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475079+0000", 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2890438200000001 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475255+0000", 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000176279 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680464+0000", 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20520917499999999 2024-04-26T01:25:00.850 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680595+0000", 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013023099999999999 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086783+0000", 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40618798900000003 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.851 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:164 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 53.534508719999998, 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78562828299999998, 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.852 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297916+0000", 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-04-26T01:25:00.853 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297932+0000", 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2542e-05 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297993+0000", 2024-04-26T01:25:00.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7813e-05 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475329+0000", 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17733589199999999 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475446+0000", 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117074 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083541+0000", 2024-04-26T01:25:00.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60809543499999996 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:165 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 53.534393305999998, 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78769006399999997, 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.856 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298030+0000", 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.59e-06 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298039+0000", 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1470000000000007e-06 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298064+0000", 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5542e-05 2024-04-26T01:25:00.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298080+0000", 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5279999999999999e-05 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476021+0000", 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.177941503 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476177+0000", 2024-04-26T01:25:00.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155873 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085718+0000", 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60954112900000001 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:166 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 53.534317264999999, 2024-04-26T01:25:00.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78831037500000001, 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 166 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298106+0000", 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-06 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298113+0000", 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7570000000000004e-06 2024-04-26T01:25:00.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298128+0000", 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5e-05 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298139+0000", 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0986e-05 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476693+0000", 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17855336099999999 2024-04-26T01:25:00.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476835+0000", 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014191599999999999 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086415+0000", 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60957998499999999 2024-04-26T01:25:00.864 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 36/34 ECSubReadReply(tid=23, attrs_read=0))", 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 53.231799238999997, 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.48295446400000003, 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600622+0000", 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.600624+0000", 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6199999999999999e-06 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601039+0000", 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00041512500000000002 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.601068+0000", 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.906e-05 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.701180+0000", 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10011215699999999 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.082862+0000", 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.381681457 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083577+0000", 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071504499999999998 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:189 3.cs0 3:395c8678:::smithi18128752-25:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:25:00.868 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 52.755381653000001, 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61527076999999997, 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077044+0000", 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0570000000000003e-06 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077061+0000", 2024-04-26T01:25:00.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6616000000000002e-05 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077097+0000", 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6829000000000003e-05 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077120+0000", 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2779999999999999e-05 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086851+0000", 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097310599999999997 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.087007+0000", 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015570199999999999 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.495435+0000", 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40842789299999999 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497946+0000", 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025112519999999998 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692090+0000", 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19414421300000001 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692311+0000", 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220368 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:190 3.cs0 3:395c8678:::smithi18128752-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 52.755265516000001, 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.42078353899999998, 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 190 2024-04-26T01:25:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077156+0000", 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077158+0000", 2024-04-26T01:25:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7540000000000001e-06 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077167+0000", 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2459999999999992e-06 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077188+0000", 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1403000000000002e-05 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077357+0000", 2024-04-26T01:25:00.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000168985 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187444+0000", 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110086218 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187592+0000", 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014808599999999999 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497585+0000", 2024-04-26T01:25:00.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.309993713 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497788+0000", 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020263099999999999 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497940+0000", 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015150299999999999 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:199 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 52.755021814999999, 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41377173299999997, 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 199 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.879 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.956e-06 2024-04-26T01:25:00.880 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077413+0000", 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1056999999999999e-05 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077435+0000", 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2602000000000001e-05 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7252999999999999e-05 2024-04-26T01:25:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187721+0000", 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.110268303 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.187913+0000", 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000191845 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.491171+0000", 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30325871700000001 2024-04-26T01:25:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:200 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 52.754944831000003, 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41453494499999999, 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.883 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 200 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077477+0000", 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.364e-06 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077486+0000", 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8159999999999994e-06 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077505+0000", 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9360000000000001e-05 2024-04-26T01:25:00.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077521+0000", 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5685999999999999e-05 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188598+0000", 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.111076802 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.188834+0000", 2024-04-26T01:25:00.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023592099999999999 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.492012+0000", 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30317799600000001 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:201 3.cs0 3:3c2c778f:::smithi18128752-24:2 [read 0~393216,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 52.754881539000003, 2024-04-26T01:25:00.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41550727300000001, 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 201 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077540+0000", 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077541+0000", 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.314e-06 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077548+0000", 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8390000000000001e-06 2024-04-26T01:25:00.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077566+0000", 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7893e-05 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077581+0000", 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5290000000000001e-05 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.189776+0000", 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.112194944 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.190248+0000", 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00047181099999999999 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.493047+0000", 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.302799182 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:322 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.682600483999998, 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.707643833, 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 322 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149822+0000", 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.004e-06 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149829+0000", 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.613e-06 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149840+0000", 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1819e-05 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249944+0000", 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100103704 2024-04-26T01:25:00.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250099+0000", 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015437400000000001 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250119+0000", 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0392999999999999e-05 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352060+0000", 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101941083 2024-04-26T01:25:00.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352219+0000", 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015886700000000001 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857465+0000", 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50524597599999999 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.896 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:323 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.582438674000002, 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.70925194000000003, 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 323 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.897 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249984+0000", 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4559999999999999e-06 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249994+0000", 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0186e-05 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250012+0000", 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7836999999999999e-05 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250023+0000", 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1143e-05 2024-04-26T01:25:00.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254128+0000", 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041046069999999997 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254145+0000", 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7405999999999999e-05 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452144+0000", 2024-04-26T01:25:00.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.197998339 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452432+0000", 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028853800000000002 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.959235+0000", 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50680242799999997 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s6 47/30 ECSubWrite(tid=123, reqid=client.4589.0:325, at_version=47'28, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.375654157, 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50042386699999997, 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456767+0000", 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456768+0000", 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7599999999999997e-07 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456800+0000", 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2240000000000003e-05 2024-04-26T01:25:00.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456819+0000", 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8799000000000001e-05 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.456835+0000", 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5860000000000001e-05 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.655702+0000", 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19886722300000001 2024-04-26T01:25:00.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.655714+0000", 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.218e-05 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957130+0000", 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301416354 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957191+0000", 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0735e-05 2024-04-26T01:25:00.905 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s6 47/30 ECSubWrite(tid=125, reqid=client.4589.0:326, at_version=47'29, trim_to=0'0, roll_forward_to=43'26))", 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.371416392999997, 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.496226999, 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461005+0000", 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461007+0000", 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7689999999999999e-06 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461040+0000", 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3563999999999999e-05 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.461069+0000", 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8989e-05 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.561206+0000", 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10013631100000001 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857800+0000", 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29659451599999997 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857828+0000", 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7583e-05 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957201+0000", 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.099372708000000004 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.957232+0000", 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1559000000000001e-05 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:397 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [copy-from ver 14] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.591622903999998, 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.58707355400000005, 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.910 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 397 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240799+0000", 2024-04-26T01:25:00.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240802+0000", 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5200000000000002e-06 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240819+0000", 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6444999999999999e-05 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240855+0000", 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6702999999999997e-05 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.240873+0000", 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7900000000000001e-05 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.241008+0000", 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013451000000000001 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.241021+0000", 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3504e-05 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.245054+0000", 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004032926 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.245220+0000", 2024-04-26T01:25:00.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016579 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.553412+0000", 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30819260700000001 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.556948+0000", 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035356580000000001 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.827661+0000", 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.270712428 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:23.827872+0000", 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021156300000000001 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.920 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:712 3.fs0 3:f7cdde45:::smithi18128752-11:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:54.822622+0000", 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0097996389999997, 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.49927348999999999, 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 712 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822622+0000", 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822622+0000", 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822623+0000", 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.004e-06 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822631+0000", 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4630000000000001e-06 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822647+0000", 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5557e-05 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822657+0000", 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.058e-05 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.922941+0000", 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10028379799999999 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.922950+0000", 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1160000000000005e-06 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:55.118810+0000", 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19586010200000001 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:55.118967+0000", 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015654699999999999 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:55.321895+0000", 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20292832299999999 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:713 3.fs0 3:f7cdde45:::smithi18128752-11:head [read 0~180224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83)", 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:54.822672+0000", 2024-04-26T01:25:00.925 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.0097496499999998, 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.59979347599999999, 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 713 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822672+0000", 2024-04-26T01:25:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822672+0000", 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822673+0000", 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6199999999999997e-07 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822678+0000", 2024-04-26T01:25:00.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6459999999999998e-06 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822695+0000", 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6886000000000001e-05 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.822705+0000", 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9899999999999992e-06 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.922992+0000", 2024-04-26T01:25:00.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10028685399999999 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:54.923000+0000", 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7970000000000001e-06 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:55.120021+0000", 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19702082400000001 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:00.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:55.120200+0000", 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017950999999999999 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:55.422465+0000", 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30226520699999998 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:00.930 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:00.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:25:00.932 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:00.931+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:01.033 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:01.036+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:01.059 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:01.059 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:01.059 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:25:01.059 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:01.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 7 to smithi18128752-33 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 33 updating version 0 to 80 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 33 version 80 is already newer than 79 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 33 version 80 is already newer than 77 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 33 version 80 is already newer than 75 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 33 version 80 is already newer than 76 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 33 version 80 is already newer than 78 2024-04-26T01:25:01.116 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 80 (ObjNum 119 snap 26 seq_num 119) dirty exists 2024-04-26T01:25:01.117 INFO:tasks.rados.rados.0.smithi181.stdout:316: left oid 33 (ObjNum 119 snap 26 seq_num 119) 2024-04-26T01:25:01.134 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:01.135+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:01.235 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:01.239+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:01.236 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:01.236 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:01.237 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:25:01.237 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:25:01.237 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:01.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:25:01.336 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:01.339+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:01.373 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:01.373 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 47/30 ECSubWrite(tid=74, reqid=client.4589.0:332, at_version=47'15, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 42.801256885999997, 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20225293799999999, 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.374 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.375 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575170+0000", 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.08e-06 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575220+0000", 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0246e-05 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575252+0000", 2024-04-26T01:25:01.376 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1482000000000001e-05 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.675393+0000", 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014118599999999 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775690+0000", 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029639899999999 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775706+0000", 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6133e-05 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777345+0000", 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016393899999999999 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777421+0000", 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6021999999999998e-05 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.378 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=58, reqid=client.4589.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.776763928000001, 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301426421, 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.379 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599662+0000", 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.299e-06 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.380 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599675+0000", 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2605e-05 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599692+0000", 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7221999999999999e-05 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.381 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699796+0000", 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100104162 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699955+0000", 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000158319 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699983+0000", 2024-04-26T01:25:01.382 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8453e-05 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901051+0000", 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201067725 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901088+0000", 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6636e-05 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=60, reqid=osd.5.0:60, at_version=48'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.676480511999998, 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20186222600000001, 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.384 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9200000000000002e-07 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.385 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699973+0000", 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8328999999999999e-05 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699980+0000", 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0770000000000002e-06 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699990+0000", 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.011e-05 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.386 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900620+0000", 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20062928899999999 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900633+0000", 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2994e-05 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901770+0000", 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001137482 2024-04-26T01:25:01.387 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901807+0000", 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6653000000000003e-05 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=146, reqid=osd.14.0:146, at_version=67'60, trim_to=0'0, roll_forward_to=61'58))", 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.585216449000001, 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200985516, 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.388 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.389 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3e-07 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791259+0000", 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9898999999999998e-05 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791388+0000", 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012909999999999999 2024-04-26T01:25:01.390 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.891494+0000", 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010556600000001 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.990490+0000", 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098996585999999998 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.990520+0000", 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0076000000000001e-05 2024-04-26T01:25:01.391 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.992124+0000", 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016037709999999999 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.992194+0000", 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9888000000000006e-05 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.392 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 67/44 ECSubWrite(tid=113, reqid=osd.5.0:113, at_version=67'78, trim_to=0'0, roll_forward_to=67'76))", 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.236522286, 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201983458, 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.393 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139905+0000", 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.86e-06 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139946+0000", 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0948999999999997e-05 2024-04-26T01:25:01.394 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139975+0000", 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9130000000000001e-05 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139990+0000", 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5712000000000001e-05 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.238557+0000", 2024-04-26T01:25:01.395 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098566661999999999 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.238587+0000", 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9757e-05 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341833+0000", 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103246136 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341886+0000", 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3251999999999998e-05 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 67/44 ECSubWrite(tid=114, reqid=osd.5.0:114, at_version=67'80, trim_to=0'0, roll_forward_to=67'76))", 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.234289788000002, 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199785569, 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.398 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142137+0000", 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.843e-06 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142182+0000", 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4796000000000003e-05 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142211+0000", 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9394999999999999e-05 2024-04-26T01:25:01.399 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142227+0000", 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5630000000000001e-05 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239528+0000", 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097300638999999994 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.400 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239543+0000", 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5616000000000001e-05 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341893+0000", 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102349381 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341921+0000", 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8269e-05 2024-04-26T01:25:01.401 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=150, reqid=osd.14.0:147, at_version=0'0, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.180588674999999, 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.146116522, 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.402 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.403 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195838+0000", 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.03e-06 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195859+0000", 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0988999999999998e-05 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195891+0000", 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1962999999999999e-05 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.404 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195911+0000", 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0046000000000001e-05 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239504+0000", 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043592858999999998 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239529+0000", 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5086999999999999e-05 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.405 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341923+0000", 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102394075 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341953+0000", 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9473000000000001e-05 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.406 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=67'64, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.178587092000001, 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14414429500000001, 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:25:01.407 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197840+0000", 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7269999999999999e-06 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.408 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197895+0000", 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5092e-05 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197934+0000", 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8846e-05 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197958+0000", 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3989e-05 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.409 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.340310+0000", 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.142352647 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.340336+0000", 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5992000000000001e-05 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341958+0000", 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016213429999999999 2024-04-26T01:25:01.410 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341982+0000", 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4658999999999999e-05 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=152, reqid=osd.14.0:152, at_version=67'66, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.176918471, 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.142969076, 2024-04-26T01:25:01.411 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.412 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199509+0000", 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1270000000000001e-06 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199563+0000", 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4565999999999999e-05 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199599+0000", 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6168999999999999e-05 2024-04-26T01:25:01.413 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199623+0000", 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3275e-05 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341198+0000", 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14157494500000001 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341226+0000", 2024-04-26T01:25:01.414 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.342404+0000", 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011773689999999999 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.342476+0000", 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1995000000000004e-05 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.415 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s3 67/63 ECSubWrite(tid=267, reqid=osd.13.0:265, at_version=0'0, trim_to=0'0, roll_forward_to=67'7))", 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.673855640999999, 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20128004799999999, 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.416 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702572+0000", 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.52e-06 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702594+0000", 2024-04-26T01:25:01.417 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2166000000000001e-05 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702623+0000", 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8960000000000001e-05 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.802711+0000", 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100088178 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.418 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.902926+0000", 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10021514500000001 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.902945+0000", 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8510000000000001e-05 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.903760+0000", 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081536499999999997 2024-04-26T01:25:01.419 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.903849+0000", 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9203999999999995e-05 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=156, reqid=osd.14.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.575722322000001, 2024-04-26T01:25:01.420 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.302725417, 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:25:01.421 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800704+0000", 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6869999999999999e-06 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800726+0000", 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1379e-05 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.422 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800752+0000", 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6472999999999999e-05 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800766+0000", 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3441999999999999e-05 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.901012+0000", 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100246591 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.423 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.901028+0000", 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.611e-05 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103277+0000", 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20224862199999999 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103428+0000", 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151113 2024-04-26T01:25:01.424 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=157, reqid=osd.14.0:157, at_version=67'68, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.573185601999999, 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30024277599999999, 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.425 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803247+0000", 2024-04-26T01:25:01.426 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5569999999999998e-06 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803288+0000", 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0778999999999997e-05 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803309+0000", 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1342999999999999e-05 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.427 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803326+0000", 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7214000000000002e-05 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.001773+0000", 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19844687899999999 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.001789+0000", 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5449999999999999e-05 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.428 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103444+0000", 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101655572 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103482+0000", 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7982000000000003e-05 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.429 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=158, reqid=osd.14.0:158, at_version=67'70, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 21.570725752000001, 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29805379300000001, 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:25:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805700+0000", 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.105e-06 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.431 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805744+0000", 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4044e-05 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805766+0000", 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1772000000000001e-05 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805778+0000", 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.166e-05 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.432 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.102518+0000", 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.296740585 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.102534+0000", 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5571999999999999e-05 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103634+0000", 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011001979999999999 2024-04-26T01:25:01.433 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103753+0000", 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118857 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 72/44 ECSubWrite(tid=197, reqid=osd.14.0:196, at_version=0'0, trim_to=0'0, roll_forward_to=72'93))", 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.595514071, 2024-04-26T01:25:01.434 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20120590699999999, 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.435 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780912+0000", 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.373e-06 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780936+0000", 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3869000000000001e-05 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780967+0000", 2024-04-26T01:25:01.436 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0565999999999998e-05 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.881085+0000", 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100118531 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.881183+0000", 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7312999999999995e-05 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.437 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.881196+0000", 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3767999999999999e-05 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.982075+0000", 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10087816300000001 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.982117+0000", 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2324000000000003e-05 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.438 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 74/44 ECSubWrite(tid=200, reqid=client.4589.0:626, at_version=74'97, trim_to=0'0, roll_forward_to=72'95))", 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.445167108, 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202684053, 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.439 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4000000000000002e-07 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.440 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931308+0000", 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9568999999999999e-05 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931342+0000", 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4266e-05 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931368+0000", 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5409999999999999e-05 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.441 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.031641+0000", 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100273353 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.031657+0000", 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5713e-05 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133885+0000", 2024-04-26T01:25:01.442 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102228584 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133942+0000", 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6617999999999999e-05 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 74/44 ECSubWrite(tid=202, reqid=client.4589.0:627, at_version=74'98, trim_to=0'0, roll_forward_to=72'95))", 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:45.940804+0000", 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.435620618, 2024-04-26T01:25:01.443 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19317337600000001, 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940804+0000", 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940804+0000", 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.444 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940806+0000", 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6700000000000001e-06 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940842+0000", 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5775999999999998e-05 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940871+0000", 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9119e-05 2024-04-26T01:25:01.445 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940887+0000", 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5625999999999999e-05 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.033111+0000", 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092224414000000005 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.446 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.033125+0000", 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4166e-05 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133949+0000", 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10082390200000001 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133978+0000", 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8702999999999999e-05 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.447 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 74/44 ECSubWrite(tid=203, reqid=client.4589.0:628, at_version=74'99, trim_to=0'0, roll_forward_to=72'95))", 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:45.943749+0000", 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.432676517999999, 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19124211899999999, 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943749+0000", 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943749+0000", 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943750+0000", 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-06 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.449 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943807+0000", 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6759000000000003e-05 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943837+0000", 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9549000000000001e-05 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943852+0000", 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5336999999999999e-05 2024-04-26T01:25:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133244+0000", 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.189391529 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133275+0000", 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1900000000000003e-05 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.134910+0000", 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016348529999999999 2024-04-26T01:25:01.451 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.134991+0000", 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0371999999999996e-05 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 78/44 ECSubWrite(tid=228, reqid=osd.14.0:226, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.739408+0000", 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.637016750000001, 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202026329, 2024-04-26T01:25:01.452 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739408+0000", 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739408+0000", 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.453 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739410+0000", 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8929999999999999e-06 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739432+0000", 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1413e-05 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739463+0000", 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1891999999999999e-05 2024-04-26T01:25:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.839568+0000", 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010424800000001 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.939843+0000", 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100275244 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.939869+0000", 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6499000000000002e-05 2024-04-26T01:25:01.455 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.941377+0000", 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015075310000000001 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.941435+0000", 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7608999999999997e-05 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 84/44 ECSubWrite(tid=144, reqid=client.4589.0:762, at_version=84'109, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:56.241295+0000", 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.1351301889999998, 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107142218, 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241295+0000", 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241295+0000", 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241298+0000", 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1729999999999999e-06 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241615+0000", 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00031733400000000001 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241679+0000", 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3672999999999998e-05 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241705+0000", 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.599e-05 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241889+0000", 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018349600000000001 2024-04-26T01:25:01.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.241921+0000", 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2929999999999998e-05 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.348377+0000", 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10645588 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.348437+0000", 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9741999999999998e-05 2024-04-26T01:25:01.460 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 84/44 ECSubWrite(tid=250, reqid=client.4589.0:753, at_version=84'134, trim_to=0'0, roll_forward_to=82'128))", 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:56.431485+0000", 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.9449405149999999, 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10774505199999999, 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.461 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.431485+0000", 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.431485+0000", 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:01.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.431485+0000", 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.431513+0000", 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7270000000000001e-05 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.431532+0000", 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8807000000000001e-05 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.431552+0000", 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9986000000000001e-05 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.535338+0000", 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103786583 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.535354+0000", 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6002999999999999e-05 2024-04-26T01:25:01.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.539152+0000", 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0037975539999999999 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.539230+0000", 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7898999999999994e-05 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:01.465 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:01.467 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:25:01.468 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:01.471+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:01.568 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:01.571+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:01.603 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:01.604 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:01.604 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:25:01.604 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:01.618 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:25:01.669 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:01.672+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:01.745 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:01.745 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:01.745 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:25:01.745 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:25:01.745 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:01.760 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:25:01.790 INFO:teuthology.orchestra.run.smithi037.stderr:marked in osd.12. 2024-04-26T01:25:01.791 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:01.795+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:01.797 INFO:tasks.rados.rados.0.smithi181.stdout:316: done (1 left) 2024-04-26T01:25:01.797 INFO:tasks.rados.rados.0.smithi181.stdout:319: done (0 left) 2024-04-26T01:25:01.797 INFO:tasks.rados.rados.0.smithi181.stdout:320: read oid 15 snap -1 2024-04-26T01:25:01.797 INFO:tasks.rados.rados.0.smithi181.stdout:320: expect (ObjNum 113 snap 24 seq_num 113) 2024-04-26T01:25:01.797 INFO:tasks.rados.rados.0.smithi181.stdout:321: snap_create 2024-04-26T01:25:01.817 INFO:tasks.thrashosds.thrasher:Added osd 12 2024-04-26T01:25:01.870 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:01.875+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:01.892 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:01.893 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:25:01.893 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:25:01.893 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:01.893 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.893 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:116 3.6s0 3:69f31b93:::smithi18128752-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:01.893 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:25:01.893 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 56.508898485000003, 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.37357116200000001, 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 116 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.894 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386857+0000", 2024-04-26T01:25:01.895 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.703e-06 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386866+0000", 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5670000000000001e-06 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386887+0000", 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1316000000000001e-05 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386904+0000", 2024-04-26T01:25:01.896 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6767000000000001e-05 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.556964+0000", 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17005975700000001 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.557176+0000", 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021186600000000001 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.897 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558110+0000", 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00093395700000000002 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.559561+0000", 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014508430000000001 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.898 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.759894+0000", 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033284600000001 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.760426+0000", 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00053253999999999997 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:117 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:25:01.899 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 56.508823831999997, 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77605625899999997, 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 117 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:25:01.900 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386931+0000", 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3599999999999999e-06 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.901 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386939+0000", 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8369999999999997e-06 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386957+0000", 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8683000000000001e-05 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386974+0000", 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.902 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558747+0000", 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17177335099999999 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558933+0000", 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018625099999999999 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761103+0000", 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202169231 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.903 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761231+0000", 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000128734 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162986+0000", 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.401754422 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:156 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~475136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:01.904 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 55.398118089, 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33900037700000002, 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 156 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.905 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497638+0000", 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2110000000000001e-06 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.906 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498467+0000", 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082923399999999998 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498486+0000", 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9341e-05 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498496+0000", 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8179999999999999e-06 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.907 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498610+0000", 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000114259 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498770+0000", 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016017599999999999 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726262+0000", 2024-04-26T01:25:01.908 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22749161900000001 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726429+0000", 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000166825 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.728061+0000", 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016325490000000001 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.909 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735773+0000", 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0077114189999999997 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836407+0000", 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10063380399999999 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836636+0000", 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022912200000000001 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.910 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:157 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 55.391897622999998, 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.43381848000000001, 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 157 2024-04-26T01:25:01.911 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.912 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503857+0000", 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2920000000000001e-06 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504200+0000", 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034286599999999999 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504218+0000", 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7961e-05 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.913 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504227+0000", 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6240000000000008e-06 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504381+0000", 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015489900000000001 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504555+0000", 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017342900000000001 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.914 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.729729+0000", 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225174034 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.730038+0000", 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030876200000000002 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.732244+0000", 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022065190000000001 2024-04-26T01:25:01.915 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735776+0000", 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035319330000000001 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937311+0000", 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201534612 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937674+0000", 2024-04-26T01:25:01.916 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00036354900000000001 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:158 3.6s0 3:60f0095a:::smithi18128752-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 55.391506550999999, 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.630196439, 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.917 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 158 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.918 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504255+0000", 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0989999999999999e-06 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504262+0000", 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5660000000000003e-06 2024-04-26T01:25:01.919 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504266+0000", 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0980000000000004e-06 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504650+0000", 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00038378400000000002 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504796+0000", 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014592199999999999 2024-04-26T01:25:01.920 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733685+0000", 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22888902699999999 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733975+0000", 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029027399999999998 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.734990+0000", 2024-04-26T01:25:01.921 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010148500000000001 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836708+0000", 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10171771 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134326+0000", 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29761846199999997 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.922 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134443+0000", 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117142 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:159 3.6s0 3:60f0095a:::smithi18128752-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 55.391480926, 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.73307043299999997, 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:01.923 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 159 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.924 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504273+0000", 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3500000000000002e-07 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504274+0000", 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.562e-06 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504281+0000", 2024-04-26T01:25:01.925 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3790000000000003e-06 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504286+0000", 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1989999999999997e-06 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504861+0000", 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00057502400000000004 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.926 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504981+0000", 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012016899999999999 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837238+0000", 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33225718900000001 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837352+0000", 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113583 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.927 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134907+0000", 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29755462599999999 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.135101+0000", 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019402500000000001 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.237343+0000", 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102242342 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.928 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=11, last_complete=35'7, committed=1, applied=1))", 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 55.061606859999998, 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30069599499999999, 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.929 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0800000000000003e-07 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834153+0000", 2024-04-26T01:25:01.930 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1699999999999996e-06 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834168+0000", 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5017999999999999e-05 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.934244+0000", 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100075936 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134251+0000", 2024-04-26T01:25:01.931 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200007516 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134843+0000", 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059144700000000004 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:171 3.6s0 3:69f31b93:::smithi18128752-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 54.559057989000003, 2024-04-26T01:25:01.932 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36489432300000002, 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 171 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.933 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336697+0000", 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3829999999999999e-06 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.934 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336701+0000", 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.053e-06 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336710+0000", 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8349999999999993e-06 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336715+0000", 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5219999999999998e-06 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.935 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337072+0000", 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00035695799999999999 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337245+0000", 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017283899999999999 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.338075+0000", 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082985399999999999 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.936 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.339323+0000", 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012477390000000001 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701394+0000", 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36207125499999998 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701590+0000", 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000195885 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.937 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 51/44 ECSubWrite(tid=71, reqid=client.4589.0:385, at_version=0'0, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 38.169344537999997, 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30170434200000001, 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.938 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726411+0000", 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.88e-06 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726432+0000", 2024-04-26T01:25:01.939 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.137e-05 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726464+0000", 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1664e-05 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826555+0000", 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100091667 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826667+0000", 2024-04-26T01:25:01.940 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011122500000000001 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826683+0000", 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6149000000000001e-05 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028057+0000", 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20137464699999999 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.941 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028113+0000", 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5739999999999998e-05 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:431 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 34.094478160000001, 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40905669700000002, 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:01.942 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 431 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:25:01.943 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801276+0000", 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1399999999999995e-07 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801283+0000", 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4840000000000001e-06 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801299+0000", 2024-04-26T01:25:01.944 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6707000000000001e-05 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801309+0000", 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0112000000000001e-05 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801486+0000", 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017675199999999999 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.945 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801494+0000", 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6490000000000008e-06 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906250+0000", 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10475599300000001 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906413+0000", 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016362200000000001 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.946 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.210332+0000", 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303918464 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:432 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 34.094370232000003, 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51044518999999999, 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.947 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 432 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.948 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801384+0000", 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801390+0000", 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2750000000000001e-06 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.949 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801402+0000", 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1747e-05 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.901507+0000", 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100104586 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.007942+0000", 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10643573100000001 2024-04-26T01:25:01.950 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.008127+0000", 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000185007 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.311828+0000", 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303700894 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.951 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 33.886299117999997, 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30115782000000002, 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.952 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009456+0000", 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7799999999999999e-06 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010014+0000", 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00055842399999999996 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.953 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010048+0000", 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3407e-05 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010060+0000", 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.254e-05 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310534+0000", 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30047343599999998 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.954 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310612+0000", 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8232999999999994e-05 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 33.883356055999997, 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.298319318, 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.955 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.956 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012397+0000", 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.957 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012425+0000", 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7229999999999998e-05 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.012432+0000", 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9160000000000001e-06 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.958 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310652+0000", 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29821998799999999 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310717+0000", 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4183999999999994e-05 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:25:01.959 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 33.867090714, 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.28320577800000002, 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028663+0000", 2024-04-26T01:25:01.960 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028664+0000", 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.266e-06 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028857+0000", 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019285299999999999 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.028876+0000", 2024-04-26T01:25:01.961 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9584000000000001e-05 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.128964+0000", 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100087615 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310808+0000", 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18184441400000001 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.962 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.311868+0000", 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001060046 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 71/44 ECSubWrite(tid=122, reqid=osd.5.0:122, at_version=71'86, trim_to=0'0, roll_forward_to=70'84))", 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 18.874194474999999, 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30196446599999999, 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.963 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.964 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.29e-07 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021579+0000", 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0132e-05 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021590+0000", 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0645000000000001e-05 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.965 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.121747+0000", 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100156917 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.322084+0000", 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033731900000001 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.322100+0000", 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5780000000000001e-05 2024-04-26T01:25:01.966 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.323445+0000", 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013450269999999999 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.323523+0000", 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8417000000000005e-05 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.967 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:603 3.6s0 3:69f31b93:::smithi18128752-39:head [stat,delete] snapc 12=[12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 18.537884299000002, 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40426880399999998, 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 603 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.968 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357872+0000", 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.207e-06 2024-04-26T01:25:01.969 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357887+0000", 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4443e-05 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357921+0000", 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3954000000000001e-05 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357937+0000", 2024-04-26T01:25:01.970 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5823000000000001e-05 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.358088+0000", 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015174899999999999 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.358258+0000", 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169999 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.971 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.359596+0000", 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001337378 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.361439+0000", 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0018429900000000001 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.761928+0000", 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.400489284 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.972 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.762138+0000", 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000209977 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s3 71/30 ECSubWrite(tid=257, reqid=osd.4.0:257, at_version=71'80, trim_to=0'0, roll_forward_to=71'76))", 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 18.136355425000001, 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30249526599999998, 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.973 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.974 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759399+0000", 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2999999999999999e-07 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759444+0000", 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5158999999999999e-05 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759467+0000", 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2892999999999999e-05 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.975 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759485+0000", 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8369999999999999e-05 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.960407+0000", 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200921553 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.960423+0000", 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.976 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.061786+0000", 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101363014 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.061893+0000", 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010717699999999999 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.977 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/44 ECSubWrite(tid=144, reqid=client.4589.0:762, at_version=84'109, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.233496+0000", 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6622568839999996, 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30456070200000002, 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233496+0000", 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.978 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233496+0000", 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233498+0000", 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9800000000000001e-06 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233738+0000", 2024-04-26T01:25:01.979 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023948599999999999 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233776+0000", 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7993e-05 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233799+0000", 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3309999999999999e-05 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.980 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.333994+0000", 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10019452199999999 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.334009+0000", 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5624000000000002e-05 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.537978+0000", 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20396856099999999 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.981 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538057+0000", 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9226000000000003e-05 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/44 ECSubWrite(tid=145, reqid=client.4589.0:763, at_version=84'110, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.235255+0000", 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6604984409999997, 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30286903199999998, 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.982 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235255+0000", 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235255+0000", 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.983 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235257+0000", 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6330000000000001e-06 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235493+0000", 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000236333 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235523+0000", 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9810000000000001e-05 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.984 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235538+0000", 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.485e-05 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.436361+0000", 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20082362000000001 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.436376+0000", 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5277e-05 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.985 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538071+0000", 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101694746 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538124+0000", 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2763000000000003e-05 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/44 ECSubWrite(tid=146, reqid=client.4589.0:764, at_version=84'111, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:01.986 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.236721+0000", 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6590320470000002, 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30146896499999998, 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236721+0000", 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.987 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236721+0000", 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236723+0000", 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8729999999999999e-06 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237026+0000", 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000302849 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.988 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237061+0000", 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5006999999999999e-05 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237076+0000", 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5393e-05 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.437332+0000", 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20025568299999999 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.989 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.437348+0000", 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5699999999999999e-05 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538129+0000", 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100781433 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538190+0000", 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1026999999999997e-05 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.990 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.991 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.991 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:01.991 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:01.991 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:01.992 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:25:01.993 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:01.995+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:02.093 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:02.096+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:02.103 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:25:02.169 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:02.169 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:02.170 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:25:02.170 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:02.185 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:25:02.194 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:02.199+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:02.247 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:25:02.248 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:25:02.268 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:25:02.269 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:25:02.289 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:25:02.290 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:25:02.294 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:02.299+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:02.305 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:02.306 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:02.306 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:25:02.306 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:25:02.306 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:02.310 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:25:02.310 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:25:02.320 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:25:02.328 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:25:02.328 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:25:02.348 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:25:02.349 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:25:02.369 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:25:02.370 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:25:02.391 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:25:02.391 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:25:02.395 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:02.399+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:02.408 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:25:02.408 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:25:02.430 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:25:02.430 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:25:02.444 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:02.444 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:02.451 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:25:02.451 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:25:02.459 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:25:02.474 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:25:02.475 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:25:02.494 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:25:02.495 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:25:02.496 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:02.499+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:02.516 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:25:02.516 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:25:02.538 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:02.539 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:02.562 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:02.562 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:02.597 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:02.602+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:02.615 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:02.615 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:02.630 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:25:02.698 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:02.700+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:02.795 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:02.795 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:02.799 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:02.803+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:02.807 INFO:tasks.rados.rados.0.smithi181.stdout:321: done (1 left) 2024-04-26T01:25:02.807 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 16 current snap is 29 2024-04-26T01:25:02.807 INFO:tasks.rados.rados.0.smithi181.stdout:322: seq_num 120 ranges {5832704=475136,6307840=655360,6963200=622592,7585792=278528} 2024-04-26T01:25:02.810 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:25:02.814 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:25:02.891 INFO:tasks.rados.rados.0.smithi181.stdout:322: writing smithi18128752-16 from 5832704 to 6307840 tid 1 2024-04-26T01:25:02.896 INFO:tasks.rados.rados.0.smithi181.stdout:322: writing smithi18128752-16 from 6307840 to 6963200 tid 2 2024-04-26T01:25:02.901 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:02.902+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:02.901 INFO:tasks.rados.rados.0.smithi181.stdout:322: writing smithi18128752-16 from 6963200 to 7585792 tid 3 2024-04-26T01:25:02.903 INFO:tasks.rados.rados.0.smithi181.stdout:322: writing smithi18128752-16 from 7585792 to 7864320 tid 4 2024-04-26T01:25:02.903 INFO:tasks.rados.rados.0.smithi181.stdout:323: rollback oid 2 current snap is 29 2024-04-26T01:25:02.904 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 2 to 28 2024-04-26T01:25:02.904 INFO:tasks.rados.rados.0.smithi181.stdout:324: snap_remove snap 26 2024-04-26T01:25:02.987 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:02.987 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:03.001 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:03.003+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:03.003 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:25:03.102 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:03.104+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:03.175 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:03.176 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:03.190 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:25:03.203 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:03.202+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:03.304 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:03.308+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:03.318 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:03.318 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:03.333 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:25:03.405 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:03.408+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:03.459 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:03.460 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:03.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:25:03.506 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:03.510+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:03.607 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:03.608+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:03.610 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:03.610 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:03.625 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:25:03.708 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:03.712+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:03.749 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:03.750 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:03.764 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:25:03.781 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:25:03.788 INFO:tasks.rados.rados.0.smithi181.stdout:324: done (3 left) 2024-04-26T01:25:03.788 INFO:tasks.rados.rados.0.smithi181.stdout:325: read oid 47 snap 27 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:325: expect (ObjNum 97 snap 17 seq_num 97) 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:326: snap_create 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:322: finishing write tid 1 to smithi18128752-16 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:322: finishing write tid 2 to smithi18128752-16 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:323: finishing rollback tid 1 to smithi18128752-2 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 0 (ObjNum 177 snap 0 seq_num 3033269328) dirty dne 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:322: finishing write tid 3 to smithi18128752-16 2024-04-26T01:25:03.789 INFO:tasks.rados.rados.0.smithi181.stdout:322: finishing write tid 4 to smithi18128752-16 2024-04-26T01:25:03.809 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:03.810+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:03.819 INFO:tasks.rados.rados.0.smithi181.stdout:322: finishing write tid 5 to smithi18128752-16 2024-04-26T01:25:03.895 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:03.895 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:03.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:25:03.911 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:03.911+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:04.010 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:04.010+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:04.020 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:04.023+0000 7f04d62fa640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 1c after trimming was started 2024-04-26T01:25:04.020 INFO:tasks.rados.rados.0.smithi181.stdout:322: finishing write tid 6 to smithi18128752-16 2024-04-26T01:25:04.020 INFO:tasks.rados.rados.0.smithi181.stdout:322: oid 16 updating version 0 to 60 2024-04-26T01:25:04.020 INFO:tasks.rados.rados.0.smithi181.stdout:322: oid 16 version 60 is already newer than 59 2024-04-26T01:25:04.021 INFO:tasks.rados.rados.0.smithi181.stdout:322: oid 16 version 60 is already newer than 58 2024-04-26T01:25:04.021 INFO:tasks.rados.rados.0.smithi181.stdout:322: oid 16 version 60 is already newer than 57 2024-04-26T01:25:04.021 INFO:tasks.rados.rados.0.smithi181.stdout:322: oid 16 version 60 is already newer than 56 2024-04-26T01:25:04.021 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 60 (ObjNum 120 snap 29 seq_num 120) dirty exists 2024-04-26T01:25:04.021 INFO:tasks.rados.rados.0.smithi181.stdout:322: left oid 16 (ObjNum 120 snap 29 seq_num 120) 2024-04-26T01:25:04.085 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:04.085 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:04.100 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:25:04.110 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:04.112+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:04.211 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:04.210+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:04.226 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:04.226 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:04.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:25:04.312 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:04.314+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:04.363 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:04.363 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:04.378 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:25:04.413 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:04.415+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:04.498 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:04.499 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:04.509 INFO:tasks.rados.rados.0.smithi181.stdout:325: expect (ObjNum 97 snap 17 seq_num 97) 2024-04-26T01:25:04.513 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:04.515+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:04.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:25:04.588 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:25:04.613 INFO:tasks.rados.rados.0.smithi181.stdout:320: expect (ObjNum 113 snap 24 seq_num 113) 2024-04-26T01:25:04.613 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:04.616+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:04.690 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:04.690 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:04.705 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:25:04.748 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:25:04.748 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:25:04.750 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:04.755+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:04.768 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:25:04.768 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:25:04.791 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:25:04.792 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:25:04.794 INFO:tasks.rados.rados.0.smithi181.stdout:320: done (4 left) 2024-04-26T01:25:04.795 INFO:tasks.rados.rados.0.smithi181.stdout:322: done (3 left) 2024-04-26T01:25:04.795 INFO:tasks.rados.rados.0.smithi181.stdout:323: done (2 left) 2024-04-26T01:25:04.795 INFO:tasks.rados.rados.0.smithi181.stdout:325: done (1 left) 2024-04-26T01:25:04.795 INFO:tasks.rados.rados.0.smithi181.stdout:326: done (0 left) 2024-04-26T01:25:04.795 INFO:tasks.rados.rados.0.smithi181.stdout:327: read oid 23 snap -1 2024-04-26T01:25:04.795 INFO:tasks.rados.rados.0.smithi181.stdout:327: expect deleted 2024-04-26T01:25:04.795 INFO:tasks.rados.rados.0.smithi181.stdout:328: snap_remove snap 23 2024-04-26T01:25:04.814 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:25:04.814 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:25:04.815 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:04.816+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:04.826 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:04.826 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:04.842 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:25:04.916 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:25:04.916 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:25:04.916 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:04.918+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:04.943 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:25:04.943 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:25:04.972 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:25:04.972 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:25:05.001 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:25:05.001 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:25:05.016 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:05.016 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:05.016 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:05.019+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:05.026 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:25:05.027 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:25:05.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:25:05.053 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:25:05.054 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:25:05.081 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:25:05.081 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:25:05.103 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:25:05.103 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:25:05.122 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:05.127+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:05.129 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:25:05.129 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:25:05.152 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:25:05.152 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:25:05.177 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:05.177 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:05.198 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:05.198 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:05.209 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:05.209 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:25:05.209 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:25:05.209 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:25:05.210 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:05.218 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:05.220+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:05.223 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:25:05.319 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:05.323+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:05.397 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:05.397 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 68.99276888, 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61300901299999999, 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:25:05.398 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 113 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405785+0000", 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.399 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405786+0000", 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5600000000000004e-07 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405803+0000", 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7666e-05 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405814+0000", 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0747000000000001e-05 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.400 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405822+0000", 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8329999999999994e-06 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417320+0000", 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.011497772999999999 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.417564+0000", 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000244842 2024-04-26T01:25:05.401 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.418017+0000", 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045273500000000001 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522469+0000", 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104451424 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018558+0000", 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.496089267 2024-04-26T01:25:05.402 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018655+0000", 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7194000000000005e-05 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018794+0000", 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013857600000000001 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.403 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 68.992718648999997, 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61322511700000004, 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 114 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.404 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405835+0000", 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1699999999999997e-07 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.405 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405842+0000", 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8530000000000003e-06 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405852+0000", 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6830000000000003e-06 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.406 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405860+0000", 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.6159999999999996e-06 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.518995+0000", 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11313486 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519240+0000", 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000245593 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:25:05.407 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.519639+0000", 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039928400000000002 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524261+0000", 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0046215249999999996 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018869+0000", 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49460855799999998 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.408 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.018947+0000", 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7742000000000002e-05 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019060+0000", 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000112686 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:25:05.409 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 68.992677865999994, 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61349101800000005, 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 115 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:25:05.410 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.411 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.411 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:25:05.411 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.411 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.411 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.413 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.405876+0000", 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.16e-07 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406016+0000", 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013951499999999999 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406029+0000", 2024-04-26T01:25:05.414 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2819999999999999e-05 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406037+0000", 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6470000000000004e-06 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520774+0000", 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.114736465 2024-04-26T01:25:05.415 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.520941+0000", 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016684800000000001 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.521147+0000", 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020602000000000001 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524283+0000", 2024-04-26T01:25:05.416 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.003136453 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019192+0000", 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.494908399 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019283+0000", 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1893999999999998e-05 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019367+0000", 2024-04-26T01:25:05.417 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3140999999999996e-05 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 68.992497885999995, 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61359576599999999, 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.418 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 116 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.419 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406056+0000", 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6400000000000001e-07 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406062+0000", 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5160000000000002e-06 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406072+0000", 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.91e-06 2024-04-26T01:25:05.420 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406080+0000", 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7030000000000006e-06 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.522858+0000", 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.116778593 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523093+0000", 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000234763 2024-04-26T01:25:05.421 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.523479+0000", 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000385831 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525176+0000", 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0016968619999999999 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019468+0000", 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49429284899999998 2024-04-26T01:25:05.422 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019546+0000", 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.7553999999999996e-05 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019651+0000", 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000105321 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.423 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4245.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 68.99246196, 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61381219799999998, 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4245", 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.37:0/2552870528", 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 117 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.424 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406091+0000", 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406092+0000", 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.6700000000000003e-07 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.425 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406100+0000", 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4560000000000003e-06 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.406109+0000", 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4590000000000004e-06 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.506278+0000", 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10016914 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.426 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524347+0000", 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.018068623999999998 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524495+0000", 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000148276 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for subops from 2", 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.524702+0000", 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020681900000000001 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.427 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op_commit", 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:56.525680+0000", 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00097787599999999992 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_commit_rec", 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019759+0000", 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.49407962300000002 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019840+0000", 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0563999999999996e-05 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.428 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:23:57.019904+0000", 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3694000000000004e-05 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:389 3.2s0 3:40bd2c39:::smithi18128752-16:head [append 0~294912] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240184+0000", 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.158369546000003, 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.607559242, 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.429 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 389 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240184+0000", 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.430 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240184+0000", 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240186+0000", 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0360000000000001e-06 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240598+0000", 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041248800000000003 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.431 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240629+0000", 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0621999999999999e-05 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240644+0000", 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5396e-05 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240755+0000", 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011019900000000001 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.432 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240766+0000", 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1087000000000001e-05 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.440598+0000", 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19983246599999999 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.440691+0000", 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2369999999999998e-05 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.433 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441366+0000", 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067539800000000001 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.746371+0000", 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30500468200000003 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.847590+0000", 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10121896800000001 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.434 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.847743+0000", 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015353 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:390 3.2s0 3:40bd2c39:::smithi18128752-16:head [setxattr _header (56) in=7b] snapc a=[a,9,8,4] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240675+0000", 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.157878584999999, 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.607429153, 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.435 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 390 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240675+0000", 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.436 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240675+0000", 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240676+0000", 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0160000000000001e-06 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240681+0000", 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5840000000000002e-06 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.437 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240693+0000", 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1552999999999999e-05 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240702+0000", 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.772e-06 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240860+0000", 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015798099999999999 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.438 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240880+0000", 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0222000000000001e-05 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.544242+0000", 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30336164199999999 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.544417+0000", 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000175773 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.439 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.545240+0000", 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00082237499999999997 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.746503+0000", 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201262949 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848038+0000", 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101535325 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.440 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848104+0000", 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.5961000000000001e-05 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:391 3.2s0 3:40bd2c39:::smithi18128752-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240751+0000", 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 42.157802615000001, 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.811133887, 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.441 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 391 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240751+0000", 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.442 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240751+0000", 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240752+0000", 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6700000000000002e-07 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240756+0000", 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1459999999999998e-06 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.443 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240768+0000", 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2456999999999999e-05 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240777+0000", 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8859999999999993e-06 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240928+0000", 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015100399999999999 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.444 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240937+0000", 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7730000000000005e-06 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.849959+0000", 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60902194200000004 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.850122+0000", 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016275999999999999 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.445 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.051885+0000", 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201762952 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:486 3.es0 3:738de3f3:::smithi18128752-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:32.997694+0000", 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.400859732999997, 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.80693512999999994, 2024-04-26T01:25:05.446 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 486 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.447 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.997694+0000", 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.997694+0000", 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.997695+0000", 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5400000000000001e-06 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.448 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.997703+0000", 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.5970000000000003e-06 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.997723+0000", 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9966000000000002e-05 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:32.997739+0000", 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6495999999999999e-05 2024-04-26T01:25:05.449 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.007483+0000", 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0097436580000000005 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.007634+0000", 2024-04-26T01:25:05.450 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015055299999999999 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.453647+0000", 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.44601390200000002 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.453788+0000", 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014032000000000001 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.451 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:33.804629+0000", 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.35084109800000002 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:536 3.3s0 3:c5d16e9b:::smithi18128752-38:head [create,setxattr jmliR (209) in=5b,setxattr eRqQZNtlPRVjcgdjmbSmOsUSjSlU]tSoVjr`^oRN (489) in=40b,setxattr aR (209) in=2b,setxattr QP`igcZgYZ`X]j_`ObrUNcR_gQZW^mQs_T`XhhQp (10033) in=40b,setxattr V[thp]nUrl^Xata^al[oh[d`ddg`p[siljQWZ^\\X (350) in=40b,setxattr Pp[amim]nQkdfb][ca]Rhonh_mr`cpapUabYUfb\\ (9612) in=40b,setxattr jecnmafmfg__r^\\S_sU]bi]kYaplTcttSSX`\\prR (18524) in=40b,setxattr fVRh^qlag^rjpdaqVk`^r]eR`P[XUlr]]agZjWq[ (1998) in=40b,setxattr hgq\\iqsggNPTRPQok (209) in=17b,setxattr frZXQX\\isoq]gmjs`rk]PiQ\\fV[RTNPNYN]YV[mq (842) in=40b,setxattr `U`PSXddYml`lYXX[sSNnPfg`VgWbXsYnftUQ`it (558) in=40b,setxattr h\\bUdlZ]kknhkaaYeTcclaXatghTgbVrhNQblto_ (946) in=40b,setxattr XgcaedbaNamUmoa^d[dhg[arrjeikfUlToSjsbY` (877) in=40b,setxattr QXqRngrniZSrdNdQaffOsm_g_pYQoedrXoNqT[mn (561) in=40b,setxattr ][ifNPWgaZd[^RnfpRhWbfS\\^a]m[\\[ienetlUUZ (173) in=40b,setxattr mnhUacnNajpOrNecmPZT[OUj[XaWmU (209) in=30b,setxattr ZcXlR_O_^fXfTQrt`cZfOgVlmhepod\\^\\`socsW` (16348) in=40b,setxattr rZrXsckW]Y`iWQiRn (209) in=17b,setxattr peUYXiNb_UsOaXnhlX_s_[QTW_SoUaN]Wal`rkao (1997) in=40b,setxattr eNsVcjeRcj\\m]Q[VhYdSqNcmPhhR[UPR^ORSk_Vd (1823) in=40b] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.788199+0000", 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 26.610354094000002, 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.54895706700000002, 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.452 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 536 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.788199+0000", 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.788199+0000", 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.453 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.788204+0000", 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4070000000000003e-06 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.788556+0000", 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00035188500000000002 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.454 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.788611+0000", 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.4997999999999998e-05 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.888714+0000", 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10010321 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.888855+0000", 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014113999999999999 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.455 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op must wait for map", 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.888868+0000", 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3360000000000001e-05 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.094468+0000", 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.205599326 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.094800+0000", 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00033251299999999999 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.456 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.096260+0000", 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014593869999999999 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.131421+0000", 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.035160982 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.336985+0000", 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20556424600000001 2024-04-26T01:25:05.457 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.337156+0000", 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017161300000000001 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:684 3.es0 3:7915a690:::smithi18128752-22:head [append 0~655360] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:49.974965+0000", 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.423588601000001, 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.52152285099999995, 2024-04-26T01:25:05.458 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 684 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:49.974965+0000", 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.459 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:49.974965+0000", 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:49.974968+0000", 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.717e-06 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:49.976031+0000", 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010635060000000001 2024-04-26T01:25:05.460 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:49.976067+0000", 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5710000000000002e-05 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076173+0000", 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100106345 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076355+0000", 2024-04-26T01:25:05.461 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00018214899999999999 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op must wait for map", 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076383+0000", 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7327000000000001e-05 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.081699+0000", 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0053169180000000003 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.081843+0000", 2024-04-26T01:25:05.462 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014375499999999999 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.083798+0000", 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019545220000000002 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.193947+0000", 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.110149519 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.463 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.496248+0000", 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30230076700000003 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.496488+0000", 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000239616 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:685 3.es0 3:7915a690:::smithi18128752-22:head [append 0~606208] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:50.076258+0000", 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.322295385, 2024-04-26T01:25:05.464 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72555617400000005, 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 685 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076258+0000", 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.465 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076258+0000", 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076262+0000", 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.563e-06 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.466 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076780+0000", 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00051855800000000002 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076812+0000", 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1785999999999997e-05 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076826+0000", 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4273e-05 2024-04-26T01:25:05.467 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076965+0000", 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013854699999999999 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076988+0000", 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3476999999999999e-05 2024-04-26T01:25:05.468 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.185702+0000", 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.108714159 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.185951+0000", 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024897899999999998 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.188711+0000", 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027596669999999999 2024-04-26T01:25:05.469 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.294615+0000", 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105904176 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.801610+0000", 2024-04-26T01:25:05.470 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50699519900000001 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.801814+0000", 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020379 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:686 3.es0 3:7915a690:::smithi18128752-22:head [append 0~475136] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:50.076858+0000", 2024-04-26T01:25:05.471 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.321695248999999, 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72557853299999997, 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 686 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076858+0000", 2024-04-26T01:25:05.472 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076858+0000", 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.076859+0000", 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3060000000000001e-06 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.473 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077133+0000", 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00027312199999999999 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077150+0000", 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7710000000000002e-05 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077160+0000", 2024-04-26T01:25:05.474 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0053e-05 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077278+0000", 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000117378 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077295+0000", 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7430000000000001e-05 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.475 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.190458+0000", 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.113162633 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.190675+0000", 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000217569 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.192664+0000", 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0019886750000000001 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.476 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.294888+0000", 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102223721 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.802228+0000", 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50733989999999995 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.802437+0000", 2024-04-26T01:25:05.477 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00020903600000000001 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:687 3.es0 3:7915a690:::smithi18128752-22:head [append 0~81920] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:50.077184+0000", 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.321369631, 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72625318699999997, 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.478 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 687 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077184+0000", 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.479 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077184+0000", 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077185+0000", 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077240+0000", 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5682e-05 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.480 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077253+0000", 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.206e-05 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077262+0000", 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4700000000000008e-06 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.481 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077371+0000", 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010901800000000001 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077382+0000", 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1233000000000001e-05 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.697174+0000", 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61979190100000003 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.482 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.697353+0000", 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017877299999999999 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.698372+0000", 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0010190119999999999 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.802460+0000", 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10408853899999999 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.483 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.803309+0000", 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00084897800000000002 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.803437+0000", 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012754800000000001 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:688 3.es0 3:7915a690:::smithi18128752-22:head [setxattr _header (57) in=7b] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:05.484 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:50.077280+0000", 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.321273343, 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.82478841199999997, 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 688 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077280+0000", 2024-04-26T01:25:05.485 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077280+0000", 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077281+0000", 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1299999999999999e-07 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.486 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077286+0000", 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.1139999999999997e-06 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077296+0000", 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9329999999999996e-06 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077304+0000", 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9929999999999995e-06 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.487 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077481+0000", 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000177166 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077500+0000", 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8470000000000001e-05 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.799656+0000", 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72215674399999996 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.488 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.799847+0000", 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00019055 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.800586+0000", 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00073953099999999998 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.802741+0000", 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0021547139999999998 2024-04-26T01:25:05.489 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.901924+0000", 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.099183335999999997 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.902068+0000", 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000144048 2024-04-26T01:25:05.490 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:689 3.es0 3:7915a690:::smithi18128752-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:50.077318+0000", 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 15.321235074000001, 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.028338365, 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.491 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 689 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077318+0000", 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077318+0000", 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.492 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077319+0000", 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9999999999999996e-07 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077323+0000", 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5300000000000001e-06 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077333+0000", 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0669999999999999e-05 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077342+0000", 2024-04-26T01:25:05.493 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5730000000000008e-06 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077607+0000", 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026500500000000001 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.077627+0000", 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0290000000000001e-05 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.494 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.804024+0000", 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.72639720900000004 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.804174+0000", 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000149517 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.903099+0000", 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098925456999999994 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.495 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.903367+0000", 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00026778199999999999 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:51.105657+0000", 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20228953199999999 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:736 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~688128] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:05.496 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.962002+0000", 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.4365510599999993, 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.56940842899999999, 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 736 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.497 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.962002+0000", 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.962002+0000", 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.962004+0000", 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.04e-06 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.498 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.962901+0000", 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00089703199999999997 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.962940+0000", 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.8828999999999997e-05 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.962966+0000", 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5253e-05 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.499 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.117170+0000", 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15420428 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.117290+0000", 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011992199999999999 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.500 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.118196+0000", 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00090638800000000005 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.221699+0000", 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103502621 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.501 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.531341+0000", 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30964207199999999 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.531411+0000", 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9992000000000003e-05 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.502 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.503 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.503 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.503 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:737 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~475136] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:05.503 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.966605+0000", 2024-04-26T01:25:05.504 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.4319479869999991, 2024-04-26T01:25:05.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.66613299599999998, 2024-04-26T01:25:05.504 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.504 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.504 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.504 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 737 2024-04-26T01:25:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.966605+0000", 2024-04-26T01:25:05.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.506 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.966605+0000", 2024-04-26T01:25:05.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.966609+0000", 2024-04-26T01:25:05.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.286e-06 2024-04-26T01:25:05.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967357+0000", 2024-04-26T01:25:05.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00074852199999999995 2024-04-26T01:25:05.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967390+0000", 2024-04-26T01:25:05.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2286000000000002e-05 2024-04-26T01:25:05.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.510 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967413+0000", 2024-04-26T01:25:05.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3623000000000001e-05 2024-04-26T01:25:05.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.219064+0000", 2024-04-26T01:25:05.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25165052599999999 2024-04-26T01:25:05.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.219162+0000", 2024-04-26T01:25:05.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8098999999999997e-05 2024-04-26T01:25:05.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.220479+0000", 2024-04-26T01:25:05.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013170289999999999 2024-04-26T01:25:05.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.223228+0000", 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0027495950000000001 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.513 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.632616+0000", 2024-04-26T01:25:05.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40938777599999998 2024-04-26T01:25:05.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.632738+0000", 2024-04-26T01:25:05.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000122254 2024-04-26T01:25:05.515 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.515 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.515 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.515 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:738 3.es0 3:734f3d85:::smithi18128752-49:head [setxattr _header (57) in=7b] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:05.515 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.967457+0000", 2024-04-26T01:25:05.516 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.4310959309999998, 2024-04-26T01:25:05.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.76574295699999995, 2024-04-26T01:25:05.516 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.516 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.516 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.516 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 738 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967457+0000", 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967457+0000", 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967459+0000", 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9269999999999999e-06 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967469+0000", 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1230000000000006e-06 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967487+0000", 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7980000000000001e-05 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967504+0000", 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7206000000000001e-05 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.428422+0000", 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.46091838099999999 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.428715+0000", 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029277200000000001 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.430122+0000", 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0014069289999999999 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:05.520 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.528537+0000", 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098414756000000006 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.733111+0000", 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20457484300000001 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.733200+0000", 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9040000000000001e-05 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.521 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:739 3.es0 3:734f3d85:::smithi18128752-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:55.967532+0000", 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 9.4310212090000007, 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.96815751900000002, 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:05.522 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 739 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967532+0000", 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967532+0000", 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:05.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967534+0000", 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3999999999999999e-06 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967540+0000", 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.2829999999999998e-06 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967561+0000", 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.069e-05 2024-04-26T01:25:05.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:55.967577+0000", 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6273000000000002e-05 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.528425+0000", 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.56084861799999997 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.528526+0000", 2024-04-26T01:25:05.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010059 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.733668+0000", 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20514175900000001 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.733803+0000", 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013546899999999999 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:05.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.935690+0000", 2024-04-26T01:25:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201886437 2024-04-26T01:25:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:05.527 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:05.527 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:05.528 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:25:05.529 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:05.530+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:05.630 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:05.630+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:05.659 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:05.659 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:05.659 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:25:05.659 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:05.673 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:25:05.730 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:05.732+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:05.797 INFO:tasks.rados.rados.0.smithi181.stdout:328: done (1 left) 2024-04-26T01:25:05.797 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 45 current snap is 30 2024-04-26T01:25:05.797 INFO:tasks.rados.rados.0.smithi181.stdout:329: seq_num 121 ranges {32768=491520,524288=802816,1327104=720896,2048000=32768} 2024-04-26T01:25:05.805 INFO:tasks.rados.rados.0.smithi181.stdout:329: writing smithi18128752-45 from 32768 to 524288 tid 1 2024-04-26T01:25:05.807 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:25:05.815 INFO:tasks.rados.rados.0.smithi181.stdout:329: writing smithi18128752-45 from 524288 to 1327104 tid 2 2024-04-26T01:25:05.824 INFO:tasks.rados.rados.0.smithi181.stdout:329: writing smithi18128752-45 from 1327104 to 2048000 tid 3 2024-04-26T01:25:05.825 INFO:tasks.rados.rados.0.smithi181.stdout:329: writing smithi18128752-45 from 2048000 to 2080768 tid 4 2024-04-26T01:25:05.825 INFO:tasks.rados.rados.0.smithi181.stdout:330: rollback oid 35 current snap is 30 2024-04-26T01:25:05.825 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 35 to 25 2024-04-26T01:25:05.825 INFO:tasks.rados.rados.0.smithi181.stdout:327: done (2 left) 2024-04-26T01:25:05.825 INFO:tasks.rados.rados.0.smithi181.stdout:331: snap_create 2024-04-26T01:25:05.831 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:05.832+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:05.847 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:05.847 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:05.847 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:25:05.847 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:25:05.847 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:05.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:25:05.932 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:05.935+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:05.989 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:05.989 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:25:05.989 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:25:05.989 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:25:05.990 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.990 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:146 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~442368,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:05.990 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:25:05.990 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.827208812000002, 2024-04-26T01:25:05.990 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.841318758, 2024-04-26T01:25:05.990 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:05.991 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:05.991 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:05.991 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:05.991 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:05.991 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 146 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:05.992 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.993 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.993 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:05.993 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165146+0000", 2024-04-26T01:25:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165148+0000", 2024-04-26T01:25:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6000000000000001e-06 2024-04-26T01:25:05.994 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.995 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:05.995 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.165979+0000", 2024-04-26T01:25:05.995 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00083094699999999996 2024-04-26T01:25:05.995 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.995 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:05.996 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166013+0000", 2024-04-26T01:25:05.996 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3386000000000002e-05 2024-04-26T01:25:05.996 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.996 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.996 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:05.997 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.166028+0000", 2024-04-26T01:25:05.997 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5480000000000001e-05 2024-04-26T01:25:05.997 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.997 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.997 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:05.997 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189275+0000", 2024-04-26T01:25:05.998 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.023246692999999999 2024-04-26T01:25:05.998 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.998 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.998 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:05.998 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.189585+0000", 2024-04-26T01:25:05.999 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00031012999999999999 2024-04-26T01:25:05.999 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:05.999 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:05.999 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:05.999 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.190986+0000", 2024-04-26T01:25:05.999 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014009820000000001 2024-04-26T01:25:06.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.000 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.000 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.000 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295619+0000", 2024-04-26T01:25:06.000 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104633299 2024-04-26T01:25:06.000 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.001 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.001 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:06.001 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005890+0000", 2024-04-26T01:25:06.001 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71027072300000005 2024-04-26T01:25:06.001 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.001 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.001 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.002 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006464+0000", 2024-04-26T01:25:06.002 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057451799999999999 2024-04-26T01:25:06.002 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.002 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.002 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.003 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.003 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.003 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:147 3.es0 3:734f3d85:::smithi18128752-49:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:06.003 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:25:06.003 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.820969368999997, 2024-04-26T01:25:06.003 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93684057600000004, 2024-04-26T01:25:06.003 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.004 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.004 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:06.004 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:06.004 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:06.005 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:06.005 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 147 2024-04-26T01:25:06.005 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.005 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.005 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.005 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.006 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:25:06.006 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.006 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.006 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.006 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.006 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171385+0000", 2024-04-26T01:25:06.007 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.007 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.007 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.007 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.007 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171387+0000", 2024-04-26T01:25:06.007 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4099999999999998e-06 2024-04-26T01:25:06.008 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.008 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.008 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.008 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.171984+0000", 2024-04-26T01:25:06.008 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00059681399999999998 2024-04-26T01:25:06.008 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.009 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.009 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172016+0000", 2024-04-26T01:25:06.009 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1779000000000002e-05 2024-04-26T01:25:06.009 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.009 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.010 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.010 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172031+0000", 2024-04-26T01:25:06.010 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5026e-05 2024-04-26T01:25:06.010 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.010 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.010 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.010 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192109+0000", 2024-04-26T01:25:06.011 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.020077514000000001 2024-04-26T01:25:06.011 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.011 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.011 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:06.011 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.192330+0000", 2024-04-26T01:25:06.012 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00022101 2024-04-26T01:25:06.012 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.012 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.012 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.012 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.193866+0000", 2024-04-26T01:25:06.012 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001536239 2024-04-26T01:25:06.013 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.013 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.013 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.013 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.295801+0000", 2024-04-26T01:25:06.013 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101934767 2024-04-26T01:25:06.013 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.014 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.014 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:06.014 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.107290+0000", 2024-04-26T01:25:06.014 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81148932200000001 2024-04-26T01:25:06.014 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.014 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.015 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.015 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108226+0000", 2024-04-26T01:25:06.015 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00093569499999999995 2024-04-26T01:25:06.015 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.015 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.015 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.016 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.016 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.016 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:148 3.es0 3:734f3d85:::smithi18128752-49:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:06.016 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:25:06.016 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.820295516999998, 2024-04-26T01:25:06.016 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.137414704, 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 148 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.017 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.018 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172059+0000", 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172060+0000", 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.513e-06 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.019 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172069+0000", 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0930000000000003e-06 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172088+0000", 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9303000000000001e-05 2024-04-26T01:25:06.020 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.021 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.021 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172103+0000", 2024-04-26T01:25:06.021 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5586e-05 2024-04-26T01:25:06.021 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.021 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496369+0000", 2024-04-26T01:25:06.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32426532299999999 2024-04-26T01:25:06.022 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.022 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:06.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.496614+0000", 2024-04-26T01:25:06.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024487400000000001 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.497656+0000", 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010425289999999999 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.599811+0000", 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10215491 2024-04-26T01:25:06.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.308958+0000", 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70914692000000001 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.309474+0000", 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00051565300000000003 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.024 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.025 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.025 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.025 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.025 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:149 3.es0 3:734f3d85:::smithi18128752-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:06.025 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:25:06.025 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.820215210000001, 2024-04-26T01:25:06.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.54710463, 2024-04-26T01:25:06.026 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.026 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.026 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:06.026 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:06.026 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 149 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172139+0000", 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172141+0000", 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.39e-06 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172148+0000", 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0600000000000002e-06 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172161+0000", 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3803e-05 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.029 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.172171+0000", 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4069999999999999e-06 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498185+0000", 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.326014476 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.498406+0000", 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000220461 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311032+0000", 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81262658200000004 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.311366+0000", 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00033330099999999999 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.719244+0000", 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40787814999999999 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:25:06.032 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.533008868000003, 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0565503999999999, 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:25:06.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459346+0000", 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9999999999999998e-07 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459365+0000", 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8600000000000001e-05 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459378+0000", 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2989e-05 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.459388+0000", 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0710000000000001e-05 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515199+0000", 2024-04-26T01:25:06.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.055811056 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515896+0000", 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00069664499999999997 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=17, last_complete=35'9, committed=1, applied=1))", 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.531427352999998, 2024-04-26T01:25:06.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54462607299999999, 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460927+0000", 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460929+0000", 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.72e-06 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460937+0000", 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.763e-06 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460965+0000", 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8693000000000001e-05 2024-04-26T01:25:06.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.460979+0000", 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4072e-05 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005512+0000", 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54453236000000005 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.005553+0000", 2024-04-26T01:25:06.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1464999999999997e-05 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=18, last_complete=35'10, committed=1, applied=1))", 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.530811880000002, 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54510908400000002, 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.040 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461543+0000", 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461544+0000", 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7370000000000001e-06 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461554+0000", 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3700000000000001e-06 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461582+0000", 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8062e-05 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.461596+0000", 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.432e-05 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006604+0000", 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.54500755300000003 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006652+0000", 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8041999999999998e-05 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.044 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=14, attrs_read=0))", 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.516933680999998, 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0397411919999999, 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475421+0000", 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475422+0000", 2024-04-26T01:25:06.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7300000000000004e-07 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475446+0000", 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4428999999999999e-05 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.475478+0000", 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1516000000000001e-05 2024-04-26T01:25:06.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.575602+0000", 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100124637 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515084+0000", 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.93948142300000004 2024-04-26T01:25:06.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515162+0000", 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8213999999999993e-05 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWriteReply(3.es0 35/34 ECSubWriteReply(tid=19, last_complete=35'11, committed=1, applied=1))", 2024-04-26T01:25:06.049 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.492932457000002, 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60899775199999995, 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499422+0000", 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499424+0000", 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7630000000000001e-06 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.052 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499436+0000", 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.201e-05 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499465+0000", 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9142999999999999e-05 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.499479+0000", 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4419e-05 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108343+0000", 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60886387600000003 2024-04-26T01:25:06.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.108420+0000", 2024-04-26T01:25:06.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6540999999999998e-05 2024-04-26T01:25:06.055 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.055 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.055 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=20, to_read={3:7f6c3760:::smithi18128752-23:head=0,4096,0}, subchunks={3:7f6c3760:::smithi18128752-23:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.391167003, 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91354819600000003, 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.056 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601187+0000", 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601264+0000", 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6395000000000003e-05 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.601293+0000", 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8829000000000001e-05 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514183+0000", 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.91289044600000002 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514736+0000", 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00055252600000000002 2024-04-26T01:25:06.061 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.390154093, 2024-04-26T01:25:06.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.113522984, 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:25:06.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602200+0000", 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602202+0000", 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602223+0000", 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1036e-05 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.602251+0000", 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7773e-05 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702368+0000", 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100117531 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715639+0000", 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0132701660000001 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.715723+0000", 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4738000000000001e-05 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:25:06.066 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.289889791999997, 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81161888599999998, 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702465+0000", 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702467+0000", 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0370000000000001e-06 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702492+0000", 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5096000000000002e-05 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702519+0000", 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7152e-05 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.702533+0000", 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4464000000000001e-05 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514000+0000", 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.81146615499999997 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.514084+0000", 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3981999999999997e-05 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.069 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=20, attrs_read=0))", 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.214354229999998, 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83750464400000002, 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778000+0000", 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778002+0000", 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.64e-06 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778026+0000", 2024-04-26T01:25:06.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4090000000000001e-05 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778058+0000", 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2398999999999999e-05 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.778076+0000", 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7408999999999999e-05 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615429+0000", 2024-04-26T01:25:06.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83735304799999999 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615505+0000", 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6057999999999996e-05 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.es0 35/34 ECSubRead(tid=21, to_read={3:7e2e4071:::smithi18128752-29:head=0,4096,0}, subchunks={3:7e2e4071:::smithi18128752-29:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.087164991999998, 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018501300000003, 2024-04-26T01:25:06.073 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905189+0000", 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905272+0000", 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2304000000000003e-05 2024-04-26T01:25:06.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.905302+0000", 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9850000000000001e-05 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615005+0000", 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70970295900000002 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.615374+0000", 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036989999999999999 2024-04-26T01:25:06.076 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.086244393000001, 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60991441499999999, 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906110+0000", 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906112+0000", 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.176e-06 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906133+0000", 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1135999999999999e-05 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906161+0000", 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7783000000000001e-05 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.906175+0000", 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3419e-05 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.515955+0000", 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60978075899999995 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.516024+0000", 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9141999999999996e-05 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 58.986200119000003, 2024-04-26T01:25:06.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71033649600000004, 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006154+0000", 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006156+0000", 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6300000000000001e-06 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006182+0000", 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6262999999999999e-05 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006213+0000", 2024-04-26T01:25:06.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0298999999999999e-05 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006233+0000", 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0169e-05 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716419+0000", 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71018600300000001 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716491+0000", 2024-04-26T01:25:06.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2132000000000007e-05 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpReadReply(3.es0 35/34 ECSubReadReply(tid=21, attrs_read=0))", 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 58.985965266000001, 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71023971900000005, 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006389+0000", 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006391+0000", 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.897e-06 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006420+0000", 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8932999999999999e-05 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006451+0000", 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1086e-05 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.006469+0000", 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8016000000000002e-05 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716554+0000", 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71008492899999998 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.716629+0000", 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4857999999999994e-05 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:254 3.es0 3:7f6c3760:::smithi18128752-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:25:06.087 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.795480836999999, 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71203804900000001, 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 254 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196874+0000", 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9299999999999997e-07 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196880+0000", 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2410000000000003e-06 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.196890+0000", 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0142999999999999e-05 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.296984+0000", 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100093949 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303480+0000", 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.006496646 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.303606+0000", 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012540699999999999 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706770+0000", 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.40316384799999999 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.706914+0000", 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000144666 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.908912+0000", 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20199735599999999 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:257 3.es0 3:7915a690:::smithi18128752-22:head [setxattr _header (56) in=7b] snapc 5=[5,4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:25:06.092 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.693825621000002, 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71064416799999996, 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 257 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298529+0000", 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298530+0000", 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7999999999999993e-07 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298537+0000", 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.8700000000000003e-06 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298551+0000", 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4202999999999999e-05 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298559+0000", 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0730000000000005e-06 2024-04-26T01:25:06.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709349+0000", 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41079035699999999 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.709598+0000", 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024883099999999998 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.710488+0000", 2024-04-26T01:25:06.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00088957200000000004 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.811226+0000", 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100738614 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.008990+0000", 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19776387300000001 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009173+0000", 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000182795 2024-04-26T01:25:06.097 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:258 3.es0 3:7915a690:::smithi18128752-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e40)", 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.693784274999999, 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.99510092800000005, 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:06.098 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 258 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298570+0000", 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298571+0000", 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298585+0000", 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4046000000000001e-05 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298599+0000", 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4192999999999999e-05 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.298608+0000", 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6030000000000005e-06 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712320+0000", 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.41371240100000001 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:11.712472+0000", 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000151586 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009806+0000", 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.297334565 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.010023+0000", 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000216698 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.293671+0000", 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.28364788600000002 2024-04-26T01:25:06.102 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.103 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.103 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.103 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.103 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.103 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:06.104 INFO:tasks.rados.rados.0.smithi181.stdout:330: finishing rollback tid 0 to smithi18128752-35 2024-04-26T01:25:06.104 INFO:tasks.rados.rados.0.smithi181.stdout:330: finishing rollback tid 1 to smithi18128752-35 2024-04-26T01:25:06.105 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:25:06.106 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:06.107+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:06.125 INFO:tasks.rados.rados.0.smithi181.stdout:330: finishing rollback tid 2 to smithi18128752-35 2024-04-26T01:25:06.125 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 8 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-04-26T01:25:06.205 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:06.207+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:06.236 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:06.237 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:06.237 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:25:06.237 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:06.252 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:25:06.305 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:06.307+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:06.369 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:06.369 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:06.369 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:25:06.369 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:25:06.369 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:06.383 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:25:06.406 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:06.411+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:06.502 INFO:tasks.rados.rados.0.smithi181.stdout:329: finishing write tid 1 to smithi18128752-45 2024-04-26T01:25:06.502 INFO:tasks.rados.rados.0.smithi181.stdout:329: finishing write tid 2 to smithi18128752-45 2024-04-26T01:25:06.503 INFO:tasks.rados.rados.0.smithi181.stdout:329: finishing write tid 3 to smithi18128752-45 2024-04-26T01:25:06.507 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:06.511+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 35/34 ECSubWrite(tid=5, reqid=client.4589.0:116, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 60.953676332999997, 2024-04-26T01:25:06.510 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20106410399999999, 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559043+0000", 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.511 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559045+0000", 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8929999999999999e-06 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559097+0000", 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2672e-05 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.559128+0000", 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1019000000000002e-05 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.512 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659233+0000", 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100104779 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659393+0000", 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016010199999999999 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.659420+0000", 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6333e-05 2024-04-26T01:25:06.513 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760047+0000", 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100627094 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:05.760107+0000", 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0211999999999999e-05 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.514 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.es5 40/34 ECSubWrite(tid=40, reqid=client.4589.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=40'23))", 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.503245118000002, 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30174757699999999, 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.515 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009474+0000", 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009476+0000", 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9800000000000001e-06 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009502+0000", 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6495999999999998e-05 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.516 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009541+0000", 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8748999999999998e-05 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.009564+0000", 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2773e-05 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109788+0000", 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100224176 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.517 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.109803+0000", 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4487e-05 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311086+0000", 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201283252 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311222+0000", 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013566399999999999 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.518 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.6s5 41/34 ECSubWrite(tid=27, reqid=osd.0.0:27, at_version=41'17, trim_to=0'0, roll_forward_to=39'15))", 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.413271588999997, 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.21182642600000001, 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.519 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099448+0000", 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.7300000000000001e-07 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.520 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099485+0000", 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7493000000000002e-05 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099509+0000", 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.387e-05 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.099527+0000", 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7598999999999999e-05 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.521 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110237+0000", 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.010710366000000001 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.110253+0000", 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5319999999999999e-05 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311234+0000", 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20098134600000001 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.522 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.311274+0000", 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.9959000000000001e-05 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=68, reqid=client.4589.0:313, at_version=46'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 48.548819547000001, 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20854388500000001, 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.523 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963900+0000", 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.524 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.963902+0000", 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5900000000000002e-06 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964264+0000", 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036215900000000003 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964306+0000", 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1801999999999997e-05 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.525 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.964334+0000", 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8133e-05 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968462+0000", 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0041272710000000001 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.526 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968486+0000", 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4219999999999999e-05 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172363+0000", 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20387691099999999 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172443+0000", 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0798999999999997e-05 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.527 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=69, reqid=client.4589.0:314, at_version=46'19, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 48.544656607, 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20440952700000001, 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:25:06.528 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968062+0000", 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968064+0000", 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9829999999999999e-06 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.529 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968174+0000", 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000109028 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968220+0000", 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6106000000000002e-05 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.968244+0000", 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4239e-05 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.530 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970652+0000", 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0024076810000000001 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970665+0000", 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3652000000000001e-05 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172450+0000", 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201785134 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.531 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172472+0000", 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1704000000000001e-05 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 46/44 ECSubWrite(tid=70, reqid=client.4589.0:315, at_version=46'20, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 48.541778450000002, 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201814353, 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.532 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970941+0000", 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.533 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.970942+0000", 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8339999999999999e-06 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971003+0000", 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.0401e-05 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971040+0000", 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7469999999999999e-05 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.534 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971064+0000", 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3609e-05 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.171968+0000", 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200904202 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.171991+0000", 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2702e-05 2024-04-26T01:25:06.535 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172718+0000", 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00072753100000000001 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.172755+0000", 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6603999999999997e-05 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.536 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 56/44 ECSubWrite(tid=107, reqid=osd.14.0:107, at_version=56'36, trim_to=0'0, roll_forward_to=55'33))", 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.363182582999997, 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20195263099999999, 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.537 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149537+0000", 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149538+0000", 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0669999999999999e-06 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149588+0000", 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0337000000000003e-05 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.538 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149623+0000", 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4782e-05 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.149641+0000", 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7886000000000001e-05 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.349947+0000", 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20030594800000001 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.539 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.349962+0000", 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5746000000000001e-05 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.351420+0000", 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0014574519999999999 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.351489+0000", 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9412999999999994e-05 2024-04-26T01:25:06.540 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds6 61/44 ECSubWrite(tid=182, reqid=osd.4.0:178, at_version=0'0, trim_to=0'0, roll_forward_to=61'33))", 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 33.110303197, 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201923306, 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.541 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402416+0000", 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402417+0000", 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.1299999999999998e-07 2024-04-26T01:25:06.542 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402437+0000", 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0322e-05 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402460+0000", 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2972999999999998e-05 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.543 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.402512+0000", 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2151999999999998e-05 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.603076+0000", 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20056403 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.603093+0000", 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6336000000000001e-05 2024-04-26T01:25:06.544 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.604291+0000", 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001198253 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.604339+0000", 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.8327e-05 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.545 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpRead(3.ds6 65/63 ECSubRead(tid=3, to_read={3:b136579b:::smithi18128752-36:4=0,262144,0}, subchunks={3:b136579b:::smithi18128752-36:4=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:37.842193+0000", 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.670526209999998, 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201211578, 2024-04-26T01:25:06.546 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842193+0000", 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.547 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842193+0000", 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842195+0000", 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.857e-06 2024-04-26T01:25:06.548 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842212+0000", 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6993000000000001e-05 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.549 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842246+0000", 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4004999999999997e-05 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:37.842264+0000", 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8320000000000001e-05 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.550 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.042660+0000", 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200396355 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.043404+0000", 2024-04-26T01:25:06.551 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00074404799999999998 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.3s4 66/63 ECSubWrite(tid=254, reqid=client.4589.0:526, at_version=66'4, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:25:06.552 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.458396492999999, 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20384141, 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:25:06.553 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:25:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.554 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054323+0000", 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9299999999999998e-07 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054927+0000", 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00060434499999999995 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.555 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.054955+0000", 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7679e-05 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.155087+0000", 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100132031 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.255298+0000", 2024-04-26T01:25:06.556 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100210753 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.255314+0000", 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5650000000000001e-05 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.557 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.258078+0000", 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0027649390000000001 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.258164+0000", 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5519999999999994e-05 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.7s1 67/30 ECSubWrite(tid=205, reqid=client.4589.0:529, at_version=0'0, trim_to=0'0, roll_forward_to=67'46))", 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.303370+0000", 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.209348813999998, 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30256664599999999, 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303370+0000", 2024-04-26T01:25:06.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303370+0000", 2024-04-26T01:25:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303372+0000", 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6169999999999999e-06 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303395+0000", 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.287e-05 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303419+0000", 2024-04-26T01:25:06.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4522e-05 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.303433+0000", 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3987e-05 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.503760+0000", 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20032639099999999 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.503792+0000", 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2066000000000001e-05 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.605860+0000", 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102068162 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.605937+0000", 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7031000000000002e-05 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.7s1 67/30 ECSubWrite(tid=206, reqid=osd.4.0:206, at_version=67'48, trim_to=0'0, roll_forward_to=67'46))", 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.338051+0000", 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.174668119, 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.26795410200000003, 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338051+0000", 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338051+0000", 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338053+0000", 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6530000000000001e-06 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338104+0000", 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.0883000000000001e-05 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.338139+0000", 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5799000000000001e-05 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.438271+0000", 2024-04-26T01:25:06.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10013135099999999 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.604310+0000", 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16603920699999999 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.604324+0000", 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3983e-05 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.605948+0000", 2024-04-26T01:25:06.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001623902 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.606005+0000", 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7324000000000002e-05 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.3s4 67/63 ECSubWrite(tid=265, reqid=osd.13.0:265, at_version=67'7, trim_to=0'0, roll_forward_to=67'5))", 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.339927+0000", 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.172792177000002, 2024-04-26T01:25:06.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.202186214, 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339927+0000", 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339927+0000", 2024-04-26T01:25:06.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339929+0000", 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.81e-06 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339966+0000", 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7085999999999997e-05 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.339995+0000", 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9581999999999999e-05 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.440127+0000", 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10013158799999999 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.540363+0000", 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10023565 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.540377+0000", 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4849e-05 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.542040+0000", 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001662771 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.542113+0000", 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.2878000000000003e-05 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.573 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 79/44 ECSubWrite(tid=236, reqid=osd.14.0:233, at_version=0'0, trim_to=0'0, roll_forward_to=79'122))", 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:51.325588+0000", 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 15.187131365999999, 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201063835, 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.325588+0000", 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.325588+0000", 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.325589+0000", 2024-04-26T01:25:06.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.747e-06 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.325611+0000", 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1129e-05 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.325639+0000", 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8636000000000002e-05 2024-04-26T01:25:06.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.325655+0000", 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5463000000000001e-05 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.425908+0000", 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10025324400000001 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.425924+0000", 2024-04-26T01:25:06.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5566e-05 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.526596+0000", 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100672154 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.526652+0000", 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.5896000000000002e-05 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.578 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 84/44 ECSubWrite(tid=248, reqid=client.4589.0:751, at_version=84'132, trim_to=0'0, roll_forward_to=82'128))", 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.235194+0000", 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.277524785000001, 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30439308199999998, 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.235194+0000", 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.235194+0000", 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.235196+0000", 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.917e-06 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.235235+0000", 2024-04-26T01:25:06.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8949000000000003e-05 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.235261+0000", 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5863e-05 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335373+0000", 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100111592 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335491+0000", 2024-04-26T01:25:06.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000117864 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335508+0000", 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7893999999999998e-05 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.539530+0000", 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20402166399999999 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.539587+0000", 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.7339000000000001e-05 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.8s6 84/44 ECSubWrite(tid=144, reqid=client.4589.0:762, at_version=84'109, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.240777+0000", 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.271941773, 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.300342832, 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.583 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.240777+0000", 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.240777+0000", 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.240779+0000", 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.240995+0000", 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00021613900000000001 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.241027+0000", 2024-04-26T01:25:06.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1296e-05 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.341126+0000", 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100099247 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.537655+0000", 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.196529118 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.586 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.537672+0000", 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7278999999999998e-05 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.541067+0000", 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0033943570000000002 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.541120+0000", 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.3646000000000001e-05 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 84/44 ECSubWrite(tid=249, reqid=client.4589.0:752, at_version=84'133, trim_to=0'0, roll_forward_to=82'128))", 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.335430+0000", 2024-04-26T01:25:06.588 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.177289455, 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20419588499999999, 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335430+0000", 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335430+0000", 2024-04-26T01:25:06.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335431+0000", 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.173e-06 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335466+0000", 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5092999999999998e-05 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335487+0000", 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0976e-05 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335500+0000", 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3447e-05 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.436772+0000", 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10127164800000001 2024-04-26T01:25:06.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.436789+0000", 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6566999999999999e-05 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.539597+0000", 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102808701 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.539626+0000", 2024-04-26T01:25:06.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.828e-05 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 84/44 ECSubWrite(tid=250, reqid=client.4589.0:753, at_version=84'134, trim_to=0'0, roll_forward_to=82'128))", 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.335523+0000", 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.17719634, 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20413060299999999, 2024-04-26T01:25:06.593 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335523+0000", 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335523+0000", 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335523+0000", 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4700000000000001e-07 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335555+0000", 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1965999999999999e-05 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335567+0000", 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.183e-05 2024-04-26T01:25:06.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.335576+0000", 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.02e-06 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.437617+0000", 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102040773 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.596 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.437643+0000", 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.567e-05 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.539628+0000", 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10198592099999999 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.539653+0000", 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4876e-05 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.597 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 94/44 ECSubWrite(tid=262, reqid=client.4589.0:819, at_version=94'144, trim_to=0'0, roll_forward_to=87'138))", 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:05.906793+0000", 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.60592579499999999, 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30320807100000002, 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.906793+0000", 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.906793+0000", 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.906795+0000", 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6670000000000001e-06 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.907022+0000", 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00022667299999999999 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.907057+0000", 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5615999999999999e-05 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.007133+0000", 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100076206 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.107333+0000", 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10019976 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.107349+0000", 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.592e-05 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.209916+0000", 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102566397 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.210001+0000", 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.5832000000000001e-05 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.0s5 94/44 ECSubWrite(tid=263, reqid=client.4589.0:820, at_version=94'145, trim_to=0'0, roll_forward_to=87'138))", 2024-04-26T01:25:06.602 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:06.007159+0000", 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.50556008500000005, 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20290341000000001, 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.007159+0000", 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.007159+0000", 2024-04-26T01:25:06.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.007160+0000", 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9299999999999998e-07 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:06.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.007293+0000", 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013338099999999999 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.007304+0000", 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.112e-05 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.107440+0000", 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100135827 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.109220+0000", 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0017802269999999999 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.109236+0000", 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6266e-05 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.210011+0000", 2024-04-26T01:25:06.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10077409399999999 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.210062+0000", 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.1901999999999999e-05 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:06.607 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:06.608 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:06.609 INFO:tasks.rados.rados.0.smithi181.stdout:329: finishing write tid 4 to smithi18128752-45 2024-04-26T01:25:06.610 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:25:06.610 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:06.612+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:06.626 INFO:tasks.rados.rados.0.smithi181.stdout:329: finishing write tid 5 to smithi18128752-45 2024-04-26T01:25:06.710 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:06.715+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:06.736 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:06.737 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:06.737 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:25:06.737 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:06.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:25:06.799 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:25:06.803 INFO:tasks.rados.rados.0.smithi181.stdout:330: done (2 left) 2024-04-26T01:25:06.804 INFO:tasks.rados.rados.0.smithi181.stdout:331: done (1 left) 2024-04-26T01:25:06.804 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 43 current snap is 31 2024-04-26T01:25:06.804 INFO:tasks.rados.rados.0.smithi181.stdout:332: seq_num 122 ranges {3391488=507904,3899392=425984,4325376=442368,4767744=540672,5308416=147456} 2024-04-26T01:25:06.811 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:06.815+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:06.818 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 7, 12] out_osds: [2] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:25:06.818 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:25:06.818 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-04-26T01:25:06.818 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 16 2024-04-26T01:25:06.819 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:25:06.854 INFO:tasks.rados.rados.0.smithi181.stdout:332: writing smithi18128752-43 from 3391488 to 3899392 tid 1 2024-04-26T01:25:06.858 INFO:tasks.rados.rados.0.smithi181.stdout:332: writing smithi18128752-43 from 3899392 to 4325376 tid 2 2024-04-26T01:25:06.863 INFO:tasks.rados.rados.0.smithi181.stdout:332: writing smithi18128752-43 from 4325376 to 4767744 tid 3 2024-04-26T01:25:06.868 INFO:tasks.rados.rados.0.smithi181.stdout:332: writing smithi18128752-43 from 4767744 to 5308416 tid 4 2024-04-26T01:25:06.869 INFO:tasks.rados.rados.0.smithi181.stdout:332: writing smithi18128752-43 from 5308416 to 5455872 tid 5 2024-04-26T01:25:06.870 INFO:tasks.rados.rados.0.smithi181.stdout:333: read oid 20 snap -1 2024-04-26T01:25:06.870 INFO:tasks.rados.rados.0.smithi181.stdout:333: expect (ObjNum 81 snap 13 seq_num 81) 2024-04-26T01:25:06.870 INFO:tasks.rados.rados.0.smithi181.stdout:334: snap_remove snap 25 2024-04-26T01:25:06.876 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:06.876 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:06.876 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:25:06.876 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:25:06.876 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:06.891 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:25:06.911 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:06.915+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:07.012 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:07.016+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:07.015 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:07.015 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:25:07.015 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/30 ECSubWrite(tid=47, reqid=client.4589.0:140, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 60.725476991000001, 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20188384300000001, 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:25:07.016 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292919+0000", 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292921+0000", 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8870000000000001e-06 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.017 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.292964+0000", 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.3152000000000003e-05 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.293004+0000", 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.0065999999999999e-05 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.393103+0000", 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100099122 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.018 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493412+0000", 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10030921299999999 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.493428+0000", 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494705+0000", 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0012777590000000001 2024-04-26T01:25:07.019 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:06.494803+0000", 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7477000000000002e-05 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 36/34 ECSubWrite(tid=26, reqid=client.4589.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=35'9))", 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.338101514999998, 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200889605, 2024-04-26T01:25:07.020 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680294+0000", 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.021 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680296+0000", 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8640000000000001e-06 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680320+0000", 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4202e-05 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.680350+0000", 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9998999999999999e-05 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.022 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780494+0000", 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100143549 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780762+0000", 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00026773600000000003 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.780776+0000", 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4705999999999999e-05 2024-04-26T01:25:07.023 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881147+0000", 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100370127 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:07.881184+0000", 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7422000000000002e-05 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.024 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 41/30 ECSubWrite(tid=90, reqid=osd.4.0:90, at_version=41'10, trim_to=0'0, roll_forward_to=37'8))", 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.818742813999997, 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103318356, 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.025 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199653+0000", 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199654+0000", 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6599999999999995e-07 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199704+0000", 2024-04-26T01:25:07.026 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.9656e-05 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199735+0000", 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1396000000000002e-05 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.199762+0000", 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7092000000000001e-05 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.299984+0000", 2024-04-26T01:25:07.027 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100221583 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.300000+0000", 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5906999999999999e-05 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302870+0000", 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028700879999999998 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.302971+0000", 2024-04-26T01:25:07.028 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010186799999999999 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 41/30 ECSubWrite(tid=55, reqid=osd.14.0:54, at_version=0'0, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 54.616353685, 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.201895835, 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.029 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402042+0000", 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402045+0000", 2024-04-26T01:25:07.030 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2800000000000002e-06 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402074+0000", 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9366e-05 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.402148+0000", 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4255e-05 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502254+0000", 2024-04-26T01:25:07.031 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100106322 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502424+0000", 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000169259 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.502449+0000", 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4909000000000002e-05 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.032 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603856+0000", 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101407649 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:12.603938+0000", 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1794999999999999e-05 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 48/34 ECSubWrite(tid=49, reqid=client.4589.0:334, at_version=48'23, trim_to=0'0, roll_forward_to=45'21))", 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:25:07.033 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 47.653546683000002, 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103332259, 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.034 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364849+0000", 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.364850+0000", 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.0299999999999998e-07 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365180+0000", 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00032962699999999998 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.035 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365243+0000", 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3516000000000006e-05 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365268+0000", 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4522e-05 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.036 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365458+0000", 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000189946 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.365484+0000", 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6800000000000001e-05 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.468084+0000", 2024-04-26T01:25:07.037 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102600071 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:19.468182+0000", 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.7073999999999999e-05 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 51/44 ECSubWrite(tid=206, reqid=client.4589.0:388, at_version=51'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:25:07.038 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.675451930999998, 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.138940597, 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342944+0000", 2024-04-26T01:25:07.039 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.040 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.040 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.040 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.342946+0000", 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343263+0000", 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000317664 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343300+0000", 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6293e-05 2024-04-26T01:25:07.041 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.343321+0000", 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1605999999999999e-05 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.479650+0000", 2024-04-26T01:25:07.042 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.136328692 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.479665+0000", 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5393999999999999e-05 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.481807+0000", 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021412390000000001 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.043 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.481885+0000", 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.7959000000000004e-05 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 51/34 ECSubWrite(tid=59, reqid=client.4589.0:397, at_version=51'30, trim_to=0'0, roll_forward_to=49'28))", 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.464807573000002, 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10403118, 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.044 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553588+0000", 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553589+0000", 2024-04-26T01:25:07.045 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6700000000000003e-07 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553663+0000", 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3734999999999998e-05 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.046 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553697+0000", 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.4003e-05 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.553722+0000", 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4678999999999999e-05 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.653970+0000", 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100247954 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.047 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.653997+0000", 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7089000000000001e-05 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.657525+0000", 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0035286039999999999 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.657620+0000", 2024-04-26T01:25:07.048 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.4349000000000001e-05 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 51/34 ECSubWrite(tid=60, reqid=client.4589.0:397, at_version=0'0, trim_to=0'0, roll_forward_to=51'30))", 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.189545541999998, 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103627764, 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.049 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828850+0000", 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.050 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828851+0000", 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6000000000000004e-07 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828874+0000", 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3206000000000001e-05 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828896+0000", 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2209999999999999e-05 2024-04-26T01:25:07.051 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.053 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.053 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.828918+0000", 2024-04-26T01:25:07.053 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1687000000000001e-05 2024-04-26T01:25:07.053 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.053 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.929363+0000", 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100445169 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.929394+0000", 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0456000000000001e-05 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932323+0000", 2024-04-26T01:25:07.054 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029296819999999999 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932478+0000", 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015479400000000001 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 52/30 ECSubWrite(tid=97, reqid=client.4589.0:400, at_version=52'20, trim_to=0'0, roll_forward_to=51'18))", 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:25:07.055 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 43.086362923000003, 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103051591, 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:25:07.056 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932033+0000", 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8599999999999999e-07 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932092+0000", 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.8335999999999998e-05 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.057 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932116+0000", 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4176e-05 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932131+0000", 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5187e-05 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932328+0000", 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019693900000000001 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.058 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:23.932354+0000", 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5783e-05 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.035001+0000", 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102647556 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:24.035085+0000", 2024-04-26T01:25:07.059 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3227999999999997e-05 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=84, reqid=osd.11.0:84, at_version=67'55, trim_to=0'0, roll_forward_to=66'51))", 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.799016+0000", 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.219380237999999, 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.203213587, 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.060 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799016+0000", 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799016+0000", 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799018+0000", 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9530000000000002e-06 2024-04-26T01:25:07.061 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799071+0000", 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.2871999999999998e-05 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799101+0000", 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0176e-05 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.062 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.799123+0000", 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2336000000000001e-05 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.000299+0000", 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20117579299999999 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.000325+0000", 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5709999999999999e-05 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.063 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.002152+0000", 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001827953 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.002229+0000", 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.6793999999999997e-05 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 67/44 ECSubWrite(tid=256, reqid=osd.13.0:256, at_version=67'41, trim_to=0'0, roll_forward_to=61'39))", 2024-04-26T01:25:07.064 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.122061197000001, 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103034636, 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.065 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896335+0000", 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3e-07 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896392+0000", 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6715000000000002e-05 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.066 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896424+0000", 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1458999999999998e-05 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.896451+0000", 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.7722999999999999e-05 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.997106+0000", 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100655038 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.067 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.997139+0000", 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2872e-05 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.999288+0000", 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021490179999999999 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.999369+0000", 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.1081000000000005e-05 2024-04-26T01:25:07.068 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 67/44 ECSubWrite(tid=258, reqid=osd.13.0:258, at_version=67'43, trim_to=0'0, roll_forward_to=61'39))", 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:38.899332+0000", 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.119063858000001, 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19960048699999999, 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.069 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899332+0000", 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899332+0000", 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899334+0000", 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4300000000000001e-06 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.070 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899371+0000", 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7863e-05 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899413+0000", 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1721999999999998e-05 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:38.899434+0000", 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0665999999999998e-05 2024-04-26T01:25:07.071 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.097162+0000", 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.19772809499999999 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.097188+0000", 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6398999999999999e-05 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.098853+0000", 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016651839999999999 2024-04-26T01:25:07.072 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.098933+0000", 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9128000000000005e-05 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=85, reqid=osd.11.0:84, at_version=0'0, trim_to=0'0, roll_forward_to=67'55))", 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.004160+0000", 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.014235546999998, 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30273594399999998, 2024-04-26T01:25:07.073 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004160+0000", 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004160+0000", 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.074 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004162+0000", 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3969999999999999e-06 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004181+0000", 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9018999999999999e-05 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004205+0000", 2024-04-26T01:25:07.075 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4553e-05 2024-04-26T01:25:07.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.076 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.076 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.076 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.004221+0000", 2024-04-26T01:25:07.076 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5282999999999999e-05 2024-04-26T01:25:07.076 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.104433+0000", 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10021221199999999 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.104464+0000", 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1556000000000001e-05 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.077 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306840+0000", 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.202375208 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306896+0000", 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6715999999999997e-05 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=86, reqid=osd.11.0:86, at_version=67'57, trim_to=0'0, roll_forward_to=67'55))", 2024-04-26T01:25:07.078 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.006383+0000", 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.012013144000001, 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30055294300000002, 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006383+0000", 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.079 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006383+0000", 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006385+0000", 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006423+0000", 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8872000000000003e-05 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.080 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006453+0000", 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9529000000000001e-05 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.006469+0000", 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5982999999999999e-05 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.205542+0000", 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.199073163 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.081 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.205561+0000", 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8777e-05 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306904+0000", 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.101343403 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306936+0000", 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1466e-05 2024-04-26T01:25:07.082 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.cs3 67/34 ECSubWrite(tid=87, reqid=osd.11.0:87, at_version=67'59, trim_to=0'0, roll_forward_to=67'55))", 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.007888+0000", 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 28.010508135999999, 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30151905099999998, 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.083 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007888+0000", 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007888+0000", 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007890+0000", 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.75e-06 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007928+0000", 2024-04-26T01:25:07.084 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.8844999999999998e-05 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007958+0000", 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9408999999999999e-05 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.007973+0000", 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5583999999999999e-05 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306254+0000", 2024-04-26T01:25:07.085 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.29828028099999998 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.306291+0000", 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.6989000000000001e-05 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.309343+0000", 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.003052758 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.309407+0000", 2024-04-26T01:25:07.086 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.3435000000000005e-05 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 67/30 ECSubWrite(tid=154, reqid=osd.14.0:154, at_version=67'35, trim_to=0'0, roll_forward_to=67'33))", 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.295910+0000", 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.722485651, 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.20202020500000001, 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.087 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295910+0000", 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295910+0000", 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295911+0000", 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0070000000000001e-06 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.088 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.295981+0000", 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9771000000000007e-05 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.296009+0000", 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8223000000000001e-05 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.396170+0000", 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10016037699999999 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.089 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.496393+0000", 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.1002237 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.496413+0000", 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9202999999999999e-05 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.497857+0000", 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001444503 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.090 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.497930+0000", 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3421000000000001e-05 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 67/30 ECSubWrite(tid=155, reqid=osd.14.0:154, at_version=0'0, trim_to=0'0, roll_forward_to=67'35))", 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 27.218560354000001, 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.200835817, 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.091 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.092 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799836+0000", 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.4300000000000002e-07 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799857+0000", 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0562999999999999e-05 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799879+0000", 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1872999999999999e-05 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.093 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.799900+0000", 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.1682e-05 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.834641+0000", 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.034740756999999997 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:39.834670+0000", 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8345999999999999e-05 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.094 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:40.000546+0000", 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.16587683 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:40.000671+0000", 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00012502300000000001 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.1s4 84/44 ECSubWrite(tid=294, reqid=client.4589.0:760, at_version=0'0, trim_to=0'0, roll_forward_to=84'41))", 2024-04-26T01:25:07.095 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.937913+0000", 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 10.080483107999999, 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.113772681, 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.937913+0000", 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.937913+0000", 2024-04-26T01:25:07.096 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.937915+0000", 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7400000000000001e-06 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.937937+0000", 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.23e-05 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.097 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.937967+0000", 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0335999999999999e-05 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:57.038109+0000", 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10014187200000001 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:57.038275+0000", 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016618000000000001 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.098 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:57.038296+0000", 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.0466999999999999e-05 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:57.051626+0000", 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.013330317 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:57.051686+0000", 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9469000000000003e-05 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.099 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.9s6 88/30 ECSubWrite(tid=257, reqid=client.4589.0:795, at_version=88'64, trim_to=0'0, roll_forward_to=82'63))", 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:00.139293+0000", 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 6.8791032169999999, 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.104457945, 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.139293+0000", 2024-04-26T01:25:07.100 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.139293+0000", 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.139294+0000", 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9300000000000006e-07 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.140401+0000", 2024-04-26T01:25:07.101 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0011074730000000001 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.140433+0000", 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1557000000000003e-05 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.240578+0000", 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100144919 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.102 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.240753+0000", 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017530599999999999 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.240770+0000", 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6857000000000001e-05 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.243657+0000", 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028873620000000001 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.103 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:00.243751+0000", 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3478000000000002e-05 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "MOSDECSubOpWrite(3.2s1 94/44 ECSubWrite(tid=342, reqid=osd.13.0:340, at_version=0'0, trim_to=0'0, roll_forward_to=94'64))", 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:06.232195+0000", 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 0.78620078699999996, 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.2004505, 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.104 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "reached pg", 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.232195+0000", 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.232195+0000", 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.232197+0000", 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4500000000000001e-06 2024-04-26T01:25:07.105 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.232222+0000", 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.491e-05 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.232241+0000", 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9973e-05 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.332363+0000", 2024-04-26T01:25:07.106 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10012170300000001 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.332496+0000", 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013297499999999999 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.332508+0000", 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1449e-05 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.432613+0000", 2024-04-26T01:25:07.107 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100105491 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.432646+0000", 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2549000000000003e-05 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.108 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:07.109 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:25:07.113 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:07.115+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:07.133 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:07.134 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:25:07.153 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-04-26T01:25:06.805550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87687712,"num_objects":58,"num_object_clones":13,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":1,"num_read":546,"num_read_kb":125594,"num_write":806,"num_write_kb":130436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":29905440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1098,"ondisk_log_size":1098,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":626592,"kb_used_data":162956,"kb_used_omap":23,"kb_used_meta":463336,"kb_avail":1414951008,"statfs":{"total":1449551462400,"available":1448909832192,"internally_reserved":0,"allocated":166866944,"data_stored":194441847,"data_compressed":3112881,"data_compressed_allocated":4276224,"data_compressed_original":9424896,"omap_allocated":23926,"internal_metadata":474456714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4210688,"num_objects":0,"num_object_clones":-1,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":11,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997149"},"pg_stats":[{"pgid":"3.f","version":"34'4","reported_seq":310,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.814485+0000","last_change":"2024-04-26T01:24:58.801255+0000","last_active":"2024-04-26T01:25:02.814485+0000","last_peered":"2024-04-26T01:25:02.814485+0000","last_clean":"2024-04-26T01:25:02.814485+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:25:02.814485+0000","last_undegraded":"2024-04-26T01:25:02.814485+0000","last_fullsized":"2024-04-26T01:25:02.814485+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.335579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016847419999999999,"stat_sum":{"num_bytes":753664,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":2212,"num_write":4,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.e","version":"87'73","reported_seq":314,"reported_epoch":91,"state":"active+degraded","last_fresh":"2024-04-26T01:25:03.422052+0000","last_change":"2024-04-26T01:25:03.422052+0000","last_active":"2024-04-26T01:25:03.422052+0000","last_peered":"2024-04-26T01:25:03.422052+0000","last_clean":"2024-04-26T01:25:00.884092+0000","last_became_active":"2024-04-26T01:25:03.422052+0000","last_became_peered":"2024-04-26T01:25:03.422052+0000","last_unstale":"2024-04-26T01:25:03.422052+0000","last_undegraded":"2024-04-26T01:25:02.921102+0000","last_fullsized":"2024-04-26T01:25:03.422052+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21431018499999999,"stat_sum":{"num_bytes":6569984,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":12431,"num_write":34,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5128192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,12],"acting":[13,9,1,11,8,6,12],"avail_no_missing":["13(0)","1(2)","6(5)","8(4)","9(1)","11(3)","14(6)"],"object_location_counts":[{"shards":"1(2),6(5),8(4),9(1),11(3),13(0),14(6)","objects":5}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.d","version":"87'72","reported_seq":215,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.823241+0000","last_change":"2024-04-26T01:24:59.226997+0000","last_active":"2024-04-26T01:25:02.823241+0000","last_peered":"2024-04-26T01:25:02.823241+0000","last_clean":"2024-04-26T01:25:02.823241+0000","last_became_active":"2024-04-26T01:24:36.105677+0000","last_became_peered":"2024-04-26T01:24:36.105677+0000","last_unstale":"2024-04-26T01:25:02.823241+0000","last_undegraded":"2024-04-26T01:25:02.823241+0000","last_fullsized":"2024-04-26T01:25:02.823241+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.029106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27407798700000002,"stat_sum":{"num_bytes":4849664,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":35,"num_read_kb":9898,"num_write":28,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.c","version":"87'88","reported_seq":398,"reported_epoch":91,"state":"active+recovering","last_fresh":"2024-04-26T01:25:03.327939+0000","last_change":"2024-04-26T01:25:03.323285+0000","last_active":"2024-04-26T01:25:03.327939+0000","last_peered":"2024-04-26T01:25:03.327939+0000","last_clean":"2024-04-26T01:25:01.061304+0000","last_became_active":"2024-04-26T01:25:03.109927+0000","last_became_peered":"2024-04-26T01:25:03.109927+0000","last_unstale":"2024-04-26T01:25:03.327939+0000","last_undegraded":"2024-04-26T01:25:03.327939+0000","last_fullsized":"2024-04-26T01:25:03.327939+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20616161,"stat_sum":{"num_bytes":7192576,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":3851,"num_write":72,"num_write_kb":11312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":["11(0)","0(2)","5(6)","6(4)","8(1)","10(3)","14(5)"],"object_location_counts":[{"shards":"0(2),5(6),6(4),8(1),10(3),11(0),14(5)","objects":5}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.b","version":"72'28","reported_seq":354,"reported_epoch":91,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-26T01:25:03.018985+0000","last_change":"2024-04-26T01:25:03.119584+0000","last_active":"2024-04-26T01:25:03.018985+0000","last_peered":"2024-04-26T01:25:03.018985+0000","last_clean":"2024-04-26T01:25:00.984918+0000","last_became_active":"2024-04-26T01:25:03.018610+0000","last_became_peered":"2024-04-26T01:25:03.018610+0000","last_unstale":"2024-04-26T01:25:03.018985+0000","last_undegraded":"2024-04-26T01:25:02.818599+0000","last_fullsized":"2024-04-26T01:25:03.018985+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30903755300000002,"stat_sum":{"num_bytes":1458176,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":573,"num_write":30,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1949696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":["4(0)","0(1)","1(2)","6(4)","7(5)","9(3)","13(6)"],"object_location_counts":[{"shards":"0(1),1(2),4(0),6(4),7(5),9(3),13(6)","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.a","version":"82'39","reported_seq":342,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:25:01.816771+0000","last_change":"2024-04-26T01:24:58.948922+0000","last_active":"2024-04-26T01:25:01.816771+0000","last_peered":"2024-04-26T01:25:01.816771+0000","last_clean":"2024-04-26T01:25:01.816771+0000","last_became_active":"2024-04-26T01:24:36.102041+0000","last_became_peered":"2024-04-26T01:24:36.102041+0000","last_unstale":"2024-04-26T01:25:01.816771+0000","last_undegraded":"2024-04-26T01:25:01.816771+0000","last_fullsized":"2024-04-26T01:25:01.816771+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.303042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011404640000000001,"stat_sum":{"num_bytes":3293184,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":9684,"num_write":33,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.9","version":"91'65","reported_seq":356,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:03.110774+0000","last_change":"2024-04-26T01:24:58.994602+0000","last_active":"2024-04-26T01:25:03.110774+0000","last_peered":"2024-04-26T01:25:03.110774+0000","last_clean":"2024-04-26T01:25:03.110774+0000","last_became_active":"2024-04-26T01:24:01.291346+0000","last_became_peered":"2024-04-26T01:24:01.291346+0000","last_unstale":"2024-04-26T01:25:03.110774+0000","last_undegraded":"2024-04-26T01:25:03.110774+0000","last_fullsized":"2024-04-26T01:25:03.110774+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:03.456479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.201699304,"stat_sum":{"num_bytes":5718016,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":1015,"num_write":24,"num_write_kb":7664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.8","version":"87'118","reported_seq":446,"reported_epoch":89,"state":"peering","last_fresh":"2024-04-26T01:25:01.261092+0000","last_change":"2024-04-26T01:25:01.807890+0000","last_active":"2024-04-26T01:25:01.807890+0000","last_peered":"2024-04-26T01:25:01.261092+0000","last_clean":"2024-04-26T01:25:01.261092+0000","last_became_active":"2024-04-26T01:24:16.447854+0000","last_became_peered":"2024-04-26T01:24:16.447854+0000","last_unstale":"2024-04-26T01:25:01.261092+0000","last_undegraded":"2024-04-26T01:25:01.261092+0000","last_fullsized":"2024-04-26T01:25:01.261092+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31276866399999997,"stat_sum":{"num_bytes":13271040,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":20739,"num_write":92,"num_write_kb":18432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4161536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,12,1,14,15,6],"acting":[5,0,12,1,14,15,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,6,9,12,14,15],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"1.0","version":"34'157","reported_seq":359,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.815706+0000","last_change":"2024-04-26T01:24:56.353548+0000","last_active":"2024-04-26T01:25:02.815706+0000","last_peered":"2024-04-26T01:25:02.815706+0000","last_clean":"2024-04-26T01:25:02.815706+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:25:02.815706+0000","last_undegraded":"2024-04-26T01:25:02.815706+0000","last_fullsized":"2024-04-26T01:25:02.815706+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:10.087788+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":137,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:25:01.807377+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:25:01.807377+0000","last_peered":"2024-04-26T01:25:01.807377+0000","last_clean":"2024-04-26T01:25:01.807377+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:01.807377+0000","last_undegraded":"2024-04-26T01:25:01.807377+0000","last_fullsized":"2024-04-26T01:25:01.807377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"91'57","reported_seq":369,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:03.208120+0000","last_change":"2024-04-26T01:24:58.991215+0000","last_active":"2024-04-26T01:25:03.208120+0000","last_peered":"2024-04-26T01:25:03.208120+0000","last_clean":"2024-04-26T01:25:03.208120+0000","last_became_active":"2024-04-26T01:24:16.348216+0000","last_became_peered":"2024-04-26T01:24:16.348216+0000","last_unstale":"2024-04-26T01:25:03.208120+0000","last_undegraded":"2024-04-26T01:25:03.208120+0000","last_fullsized":"2024-04-26T01:25:03.208120+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:24.203324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019717459999999999,"stat_sum":{"num_bytes":7372800,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":1209,"num_write":27,"num_write_kb":7920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-04-26T01:25:01.816126+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:01.816126+0000","last_peered":"2024-04-26T01:25:01.816126+0000","last_clean":"2024-04-26T01:25:01.816126+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:01.816126+0000","last_undegraded":"2024-04-26T01:25:01.816126+0000","last_fullsized":"2024-04-26T01:25:01.816126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"87'48","reported_seq":370,"reported_epoch":91,"state":"active+recovery_wait+degraded","last_fresh":"2024-04-26T01:25:03.217141+0000","last_change":"2024-04-26T01:25:03.317394+0000","last_active":"2024-04-26T01:25:03.217141+0000","last_peered":"2024-04-26T01:25:03.217141+0000","last_clean":"2024-04-26T01:25:01.084678+0000","last_became_active":"2024-04-26T01:25:03.216745+0000","last_became_peered":"2024-04-26T01:25:03.216745+0000","last_unstale":"2024-04-26T01:25:03.217141+0000","last_undegraded":"2024-04-26T01:25:02.811577+0000","last_fullsized":"2024-04-26T01:25:03.217141+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.513592194,"stat_sum":{"num_bytes":2850816,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":11803,"num_write":27,"num_write_kb":9344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,12,13],"acting":[4,11,9,5,10,12,13],"avail_no_missing":["4(0)","5(3)","9(2)","10(4)","11(1)","13(6)","14(5)"],"object_location_counts":[{"shards":"4(0),5(3),9(2),10(4),11(1),13(6),14(5)","objects":3}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.816608+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:25:02.816608+0000","last_peered":"2024-04-26T01:25:02.816608+0000","last_clean":"2024-04-26T01:25:02.816608+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:02.816608+0000","last_undegraded":"2024-04-26T01:25:02.816608+0000","last_fullsized":"2024-04-26T01:25:02.816608+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"87'138","reported_seq":465,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:03.015660+0000","last_change":"2024-04-26T01:24:59.205772+0000","last_active":"2024-04-26T01:25:03.015660+0000","last_peered":"2024-04-26T01:25:03.015660+0000","last_clean":"2024-04-26T01:25:03.015660+0000","last_became_active":"2024-04-26T01:24:16.549243+0000","last_became_peered":"2024-04-26T01:24:16.549243+0000","last_unstale":"2024-04-26T01:25:03.015660+0000","last_undegraded":"2024-04-26T01:25:03.015660+0000","last_fullsized":"2024-04-26T01:25:03.015660+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:12.147034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41302566200000002,"stat_sum":{"num_bytes":6356992,"num_objects":9,"num_object_clones":2,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":53,"num_read_kb":23877,"num_write":94,"num_write_kb":12208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":144,"reported_epoch":89,"state":"peering","last_fresh":"2024-04-26T01:25:00.883122+0000","last_change":"2024-04-26T01:25:01.816896+0000","last_active":"2024-04-26T01:25:01.816896+0000","last_peered":"2024-04-26T01:25:00.883122+0000","last_clean":"2024-04-26T01:25:00.883122+0000","last_became_active":"2024-04-26T01:24:05.175683+0000","last_became_peered":"2024-04-26T01:24:05.175683+0000","last_unstale":"2024-04-26T01:25:00.883122+0000","last_undegraded":"2024-04-26T01:25:00.883122+0000","last_fullsized":"2024-04-26T01:25:00.883122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,12],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"71'11","reported_seq":335,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.817403+0000","last_change":"2024-04-26T01:24:59.091067+0000","last_active":"2024-04-26T01:25:02.817403+0000","last_peered":"2024-04-26T01:25:02.817403+0000","last_clean":"2024-04-26T01:25:02.817403+0000","last_became_active":"2024-04-26T01:24:36.104415+0000","last_became_peered":"2024-04-26T01:24:36.104415+0000","last_unstale":"2024-04-26T01:25:02.817403+0000","last_undegraded":"2024-04-26T01:25:02.817403+0000","last_fullsized":"2024-04-26T01:25:02.817403+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:23.440699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013578970000000001,"stat_sum":{"num_bytes":901120,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":4532,"num_write":23,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,6,5,3],"acting":[13,7,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":311,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.816355+0000","last_change":"2024-04-26T01:24:58.802171+0000","last_active":"2024-04-26T01:25:02.816355+0000","last_peered":"2024-04-26T01:25:02.816355+0000","last_clean":"2024-04-26T01:25:02.816355+0000","last_became_active":"2024-04-26T01:24:36.199837+0000","last_became_peered":"2024-04-26T01:24:36.199837+0000","last_unstale":"2024-04-26T01:25:02.816355+0000","last_undegraded":"2024-04-26T01:25:02.816355+0000","last_fullsized":"2024-04-26T01:25:02.816355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:27.910021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001562711,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":138,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.910982+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:25:02.910982+0000","last_peered":"2024-04-26T01:25:02.910982+0000","last_clean":"2024-04-26T01:25:02.910982+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:02.910982+0000","last_undegraded":"2024-04-26T01:25:02.910982+0000","last_fullsized":"2024-04-26T01:25:02.910982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"88'62","reported_seq":394,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T01:25:03.933225+0000","last_change":"2024-04-26T01:25:03.934087+0000","last_active":"2024-04-26T01:25:03.933225+0000","last_peered":"2024-04-26T01:25:03.933225+0000","last_clean":"2024-04-26T01:25:03.933225+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:25:03.933225+0000","last_undegraded":"2024-04-26T01:25:03.933225+0000","last_fullsized":"2024-04-26T01:25:03.933225+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.616869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018580719999999999,"stat_sum":{"num_bytes":14499840,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":15375,"num_write":35,"num_write_kb":14640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"18","length":"1"},{"start":"1a","length":"1"},{"start":"1c","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":138,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.815365+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:25:02.815365+0000","last_peered":"2024-04-26T01:25:02.815365+0000","last_clean":"2024-04-26T01:25:02.815365+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:02.815365+0000","last_undegraded":"2024-04-26T01:25:02.815365+0000","last_fullsized":"2024-04-26T01:25:02.815365+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"87'101","reported_seq":405,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.911179+0000","last_change":"2024-04-26T01:24:59.101578+0000","last_active":"2024-04-26T01:25:02.911179+0000","last_peered":"2024-04-26T01:25:02.911179+0000","last_clean":"2024-04-26T01:25:02.911179+0000","last_became_active":"2024-04-26T01:24:01.091412+0000","last_became_peered":"2024-04-26T01:24:01.091412+0000","last_unstale":"2024-04-26T01:25:02.911179+0000","last_undegraded":"2024-04-26T01:25:02.911179+0000","last_fullsized":"2024-04-26T01:25:02.911179+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:24.525297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30844030300000003,"stat_sum":{"num_bytes":11894784,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":5632,"num_write":62,"num_write_kb":13216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T01:25:03.931572+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:25:03.931572+0000","last_peered":"2024-04-26T01:25:03.931572+0000","last_clean":"2024-04-26T01:25:03.931572+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:03.931572+0000","last_undegraded":"2024-04-26T01:25:03.931572+0000","last_fullsized":"2024-04-26T01:25:03.931572+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:10.471688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"72'37","reported_seq":366,"reported_epoch":91,"state":"activating+degraded","last_fresh":"2024-04-26T01:25:02.920269+0000","last_change":"2024-04-26T01:25:02.918154+0000","last_active":"2024-04-26T01:25:01.806878+0000","last_peered":"2024-04-26T01:25:00.797659+0000","last_clean":"2024-04-26T01:25:00.797659+0000","last_became_active":"2024-04-26T01:24:36.202766+0000","last_became_peered":"2024-04-26T01:24:36.202766+0000","last_unstale":"2024-04-26T01:25:02.920269+0000","last_undegraded":"2024-04-26T01:25:02.915658+0000","last_fullsized":"2024-04-26T01:25:02.920269+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099781900000000005,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2645,"num_write":18,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1966080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":["0(0)","1(2)","7(5)","9(6)","11(4)","13(1)","15(3)"],"object_location_counts":[{"shards":"0(0),1(2),7(5),9(6),13(1),15(3)","objects":1}],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"1"},{"start":"17","length":"2"},{"start":"1a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":139,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T01:25:02.815899+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:25:02.815899+0000","last_peered":"2024-04-26T01:25:02.815899+0000","last_clean":"2024-04-26T01:25:02.815899+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:02.815899+0000","last_undegraded":"2024-04-26T01:25:02.815899+0000","last_fullsized":"2024-04-26T01:25:02.815899+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":87097344,"num_objects":56,"num_object_clones":13,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":1,"num_read":410,"num_read_kb":125476,"num_write":603,"num_write_kb":126976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":29376512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156626944,"data_stored":192278528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":941,"ondisk_log_size":941,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":773920,"data_compressed":205286,"data_compressed_allocated":282624,"data_compressed_original":565248,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44776,"kb_used_data":13592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327064,"statfs":{"total":96636764160,"available":96590913536,"internally_reserved":0,"allocated":13918208,"data_stored":15978272,"data_compressed":205286,"data_compressed_allocated":282624,"data_compressed_original":565248,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280530,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46788,"kb_used_data":15600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325052,"statfs":{"total":96636764160,"available":96588853248,"internally_reserved":0,"allocated":15974400,"data_stored":18920955,"data_compressed":202275,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313233,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46432,"kb_used_data":15252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325408,"statfs":{"total":96636764160,"available":96589217792,"internally_reserved":0,"allocated":15618048,"data_stored":18726688,"data_compressed":205286,"data_compressed_allocated":282624,"data_compressed_original":565248,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313234,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48156,"kb_used_data":16968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323684,"statfs":{"total":96636764160,"available":96587452416,"internally_reserved":0,"allocated":17375232,"data_stored":21075451,"data_compressed":202275,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313234,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313233,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40164,"kb_used_data":8984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331676,"statfs":{"total":96636764160,"available":96595636224,"internally_reserved":0,"allocated":9199616,"data_stored":10338080,"data_compressed":205286,"data_compressed_allocated":282624,"data_compressed_original":565248,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37368,"kb_used_data":6176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334472,"statfs":{"total":96636764160,"available":96598499328,"internally_reserved":0,"allocated":6324224,"data_stored":6532845,"data_compressed":233339,"data_compressed_allocated":315392,"data_compressed_original":1073152,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313233,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42652,"kb_used_data":11464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329188,"statfs":{"total":96636764160,"available":96593088512,"internally_reserved":0,"allocated":11739136,"data_stored":13780475,"data_compressed":202275,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313234,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46716,"kb_used_data":11368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94325124,"statfs":{"total":96636764160,"available":96588926976,"internally_reserved":0,"allocated":11640832,"data_stored":13372621,"data_compressed":199284,"data_compressed_allocated":274432,"data_compressed_original":548864,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313234,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46160,"kb_used_data":14972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325680,"statfs":{"total":96636764160,"available":96589496320,"internally_reserved":0,"allocated":15331328,"data_stored":17040891,"data_compressed":202275,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313234,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30632,"kb_used_data":3604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341208,"statfs":{"total":96636764160,"available":96605396992,"internally_reserved":0,"allocated":3690496,"data_stored":3417595,"data_compressed":202275,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313234,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45372,"kb_used_data":14184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326468,"statfs":{"total":96636764160,"available":96590303232,"internally_reserved":0,"allocated":14524416,"data_stored":17536507,"data_compressed":202275,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34780,"kb_used_data":7752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337060,"statfs":{"total":96636764160,"available":96601149440,"internally_reserved":0,"allocated":7938048,"data_stored":8254971,"data_compressed":202275,"data_compressed_allocated":278528,"data_compressed_original":557056,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313235,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43684,"kb_used_data":12504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328156,"statfs":{"total":96636764160,"available":96592031744,"internally_reserved":0,"allocated":12804096,"data_stored":15105824,"data_compressed":205286,"data_compressed_allocated":282624,"data_compressed_original":565248,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313234,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44860,"kb_used_data":9512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326980,"statfs":{"total":96636764160,"available":96590827520,"internally_reserved":0,"allocated":9740288,"data_stored":13586752,"data_compressed":237903,"data_compressed_allocated":323584,"data_compressed_original":1077248,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14938112,"data_stored":18157568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14852096,"data_stored":19693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5251072,"data_stored":5251072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10702848,"data_stored":13017088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":10616832,"data_stored":12619776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14577664,"data_stored":18018304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2654208,"data_stored":2654208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12869632,"data_stored":15204352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13488128,"data_stored":16773120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6901760,"data_stored":7491584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11825152,"data_stored":14401536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8855552,"data_stored":12509184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":16338944,"data_stored":20312064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8433664,"data_stored":11304960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:25:07.154 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-04-26T01:25:07.214 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:07.214+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:07.218 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:25:07.238 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:07.239 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:07.239 INFO:teuthology.orchestra.run.smithi071.stdout: "num_ops": 0 2024-04-26T01:25:07.239 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:07.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:25:07.315 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:07.319+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:07.371 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:25:07.372 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:25:07.392 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:25:07.393 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:25:07.413 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:25:07.413 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:25:07.416 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:07.420+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:07.417 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:07.418 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [], 2024-04-26T01:25:07.418 INFO:teuthology.orchestra.run.smithi071.stdout: "complaint_time": 30, 2024-04-26T01:25:07.418 INFO:teuthology.orchestra.run.smithi071.stdout: "num_blocked_ops": 0 2024-04-26T01:25:07.418 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:07.431 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:25:07.434 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:25:07.434 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:25:07.450 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:25:07.451 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:25:07.517 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:07.518+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:07.557 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-26T01:25:07.557 INFO:teuthology.orchestra.run.smithi071.stdout: "size": 20, 2024-04-26T01:25:07.557 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 600, 2024-04-26T01:25:07.557 INFO:teuthology.orchestra.run.smithi071.stdout: "ops": [ 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:196 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.483234944000003, 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61790984400000004, 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 196 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.558 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077400+0000", 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077402+0000", 2024-04-26T01:25:07.559 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.8169999999999999e-06 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077416+0000", 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3783e-05 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077453+0000", 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.7209000000000002e-05 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077478+0000", 2024-04-26T01:25:07.560 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4168999999999998e-05 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077630+0000", 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015196599999999999 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077649+0000", 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.9262999999999998e-05 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.181790+0000", 2024-04-26T01:25:07.561 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10414122100000001 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.182072+0000", 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00028194299999999999 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.183846+0000", 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001774184 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.562 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189006+0000", 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0051592809999999999 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695163+0000", 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50615769499999996 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695310+0000", 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014631299999999999 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.563 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:197 3.9s0 3:93dca175:::smithi18128752-10:head [rollback 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.483106524, 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61806578000000001, 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 197 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.564 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.565 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077528+0000", 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077529+0000", 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3239999999999999e-06 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077537+0000", 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.0560000000000005e-06 2024-04-26T01:25:07.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077556+0000", 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8456000000000001e-05 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077573+0000", 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.6566e-05 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077807+0000", 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00023464800000000001 2024-04-26T01:25:07.567 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077822+0000", 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5206e-05 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184696+0000", 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10687403099999999 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.184837+0000", 2024-04-26T01:25:07.568 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000141113 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.186190+0000", 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001352575 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189008+0000", 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028181450000000002 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695495+0000", 2024-04-26T01:25:07.569 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50648651300000003 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695594+0000", 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9147000000000005e-05 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:198 3.9s0 3:93dca175:::smithi18128752-10:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 59.483040338999999, 2024-04-26T01:25:07.570 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.61826488800000001, 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 198 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077594+0000", 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077595+0000", 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.167e-06 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077601+0000", 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6500000000000001e-06 2024-04-26T01:25:07.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077619+0000", 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7569000000000001e-05 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077643+0000", 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.4652999999999999e-05 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.077989+0000", 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00034534799999999999 2024-04-26T01:25:07.573 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.078006+0000", 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7125999999999998e-05 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187076+0000", 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10906990799999999 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.187274+0000", 2024-04-26T01:25:07.574 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000198221 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.188283+0000", 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001008994 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.189285+0000", 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001002221 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695760+0000", 2024-04-26T01:25:07.575 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.506474914 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:08.695859+0000", 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.9117000000000007e-05 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:316 3.0s0 3:03a7f525:::smithi18128752-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 49.845097471999999, 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.86208679600000004, 2024-04-26T01:25:07.576 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 316 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.577 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715537+0000", 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715538+0000", 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.68e-07 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715540+0000", 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.413e-06 2024-04-26T01:25:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715547+0000", 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.131e-06 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.715553+0000", 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.6640000000000003e-06 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855355+0000", 2024-04-26T01:25:07.579 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.13980195500000001 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.855595+0000", 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000240265 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.971741+0000", 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.116145744 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:17.972105+0000", 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00036422899999999999 2024-04-26T01:25:07.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272430+0000", 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30032516799999998 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.272617+0000", 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018708799999999999 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:18.577624+0000", 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.305006571 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.581 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(osd.4.21:9 3.0s0 3:0c3f65fa:::smithi18128752-8:head [assert-version v29,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 38.677413219000002, 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.79000656700000005, 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883221+0000", 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883225+0000", 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.2959999999999999e-06 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883240+0000", 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4977000000000001e-05 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.583 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883273+0000", 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.3818e-05 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:28.883296+0000", 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.2282000000000001e-05 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.368761+0000", 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.48546481800000002 2024-04-26T01:25:07.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.368922+0000", 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000161039 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:29.673228+0000", 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30430633699999998 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:471 3.0s0 3:01bb5e34:::smithi18128752-45:head [setxattr _header (57) in=7b] snapc e=[e,4] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:25:07.585 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.689901679000002, 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67904865400000003, 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 471 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.586 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870733+0000", 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870734+0000", 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.124e-06 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.587 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870740+0000", 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.4530000000000001e-06 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870751+0000", 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1126e-05 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870759+0000", 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4370000000000007e-06 2024-04-26T01:25:07.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.971074+0000", 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100315001 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.971323+0000", 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024919900000000002 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.972178+0000", 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00085481999999999995 2024-04-26T01:25:07.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.976513+0000", 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.004335317 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.549610+0000", 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57309677199999998 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.549782+0000", 2024-04-26T01:25:07.590 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017140499999999999 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:472 3.0s0 3:01bb5e34:::smithi18128752-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 34.689863320000001, 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.83389453400000002, 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.591 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 472 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870771+0000", 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.592 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870772+0000", 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.3699999999999999e-07 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870776+0000", 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.416e-06 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870786+0000", 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0652999999999999e-05 2024-04-26T01:25:07.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.870795+0000", 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.4640000000000006e-06 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.974442+0000", 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.10364701900000001 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:32.974615+0000", 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017313100000000001 2024-04-26T01:25:07.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.550708+0000", 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.57609289699999999 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.550827+0000", 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000119185 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:33.704666+0000", 2024-04-26T01:25:07.595 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.15383893200000001 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:677 3.0s0 3:01bb5e34:::smithi18128752-45:head [copy-from ver 14 in=55b] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.640309616, 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.149832408, 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.596 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 677 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920325+0000", 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.597 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920326+0000", 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.432e-06 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920334+0000", 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.3699999999999997e-06 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920349+0000", 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5540999999999999e-05 2024-04-26T01:25:07.598 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920360+0000", 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0868e-05 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920475+0000", 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00011452599999999999 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "op must wait for map", 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920488+0000", 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3478999999999999e-05 2024-04-26T01:25:07.599 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.923555+0000", 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0030671539999999999 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.923806+0000", 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025074800000000002 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.745394+0000", 2024-04-26T01:25:07.600 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.82158756200000005 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.748239+0000", 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0028450989999999998 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.069994+0000", 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.32175474900000001 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:51.070157+0000", 2024-04-26T01:25:07.601 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00016388000000000001 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:678 3.0s0 3:01bb5e34:::smithi18128752-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.640255969000002, 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.82652194000000001, 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.602 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 678 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.603 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920379+0000", 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 5.9999999999999997e-07 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920382+0000", 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9299999999999999e-06 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920390+0000", 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.2300000000000008e-06 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.604 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920397+0000", 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.6080000000000001e-06 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920586+0000", 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018897900000000001 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.920597+0000", 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1467e-05 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.605 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.924206+0000", 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0036080420000000001 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:49.924340+0000", 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000134797 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.746704+0000", 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.82236340299999999 2024-04-26T01:25:07.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.746807+0000", 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000103463 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.746901+0000", 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.3420999999999999e-05 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.607 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:683 3.9s0 3:93dca175:::smithi18128752-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 17.499337089000001, 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.90844643700000005, 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 683 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.608 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061298+0000", 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7300000000000005e-07 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.609 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061302+0000", 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.5860000000000001e-06 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.061313+0000", 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0689999999999999e-05 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161394+0000", 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100081715 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.610 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.161974+0000", 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00057978700000000001 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.162126+0000", 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015176100000000001 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.765908+0000", 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.60378186599999994 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.611 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.766058+0000", 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015031300000000001 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:50.969744+0000", 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.203685846 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.612 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:725 3.0s0 3:0213ebcd:::smithi18128752-32:head [copy-from ver 11] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:55.915613+0000", 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.645021535, 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.62070122800000005, 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 725 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.613 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:55.915613+0000", 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:55.915613+0000", 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:55.915615+0000", 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.3530000000000002e-06 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.614 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:55.915627+0000", 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1586999999999999e-05 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:55.915659+0000", 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1785999999999997e-05 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:55.915677+0000", 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8525999999999998e-05 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.615 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.015080+0000", 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.099402193 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.015223+0000", 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014311400000000001 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.126887+0000", 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.111664069 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.616 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.232704+0000", 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.105817363 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.536166+0000", 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30346194700000001 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.536314+0000", 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014829 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.617 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:751 3.0s0 3:0c3f65fa:::smithi18128752-8:head [append 0~0] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.123110+0000", 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.437524133, 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.62185815499999997, 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.618 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 751 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123110+0000", 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123110+0000", 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123113+0000", 2024-04-26T01:25:07.619 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5670000000000002e-06 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123123+0000", 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0285999999999999e-05 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123154+0000", 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0187000000000001e-05 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123171+0000", 2024-04-26T01:25:07.620 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7091999999999998e-05 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.233031+0000", 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.109860664 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.233308+0000", 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00027710199999999999 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.235220+0000", 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0019112070000000001 2024-04-26T01:25:07.621 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.240476+0000", 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0052565049999999999 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.744843+0000", 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50436663100000001 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.744969+0000", 2024-04-26T01:25:07.622 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000125914 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:752 3.0s0 3:0c3f65fa:::smithi18128752-8:head [rollback 19] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.123200+0000", 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.437434844, 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.62202549100000004, 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.623 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 752 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123200+0000", 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123200+0000", 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.624 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123201+0000", 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.9599999999999998e-07 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123208+0000", 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.9670000000000004e-06 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123219+0000", 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1096e-05 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.625 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123227+0000", 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9300000000000003e-06 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.236755+0000", 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.113528636 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.236915+0000", 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00015997299999999999 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.626 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.237925+0000", 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0010094609999999999 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.240631+0000", 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002706076 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.745129+0000", 2024-04-26T01:25:07.627 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.50449843500000002 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.745225+0000", 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 9.6020999999999994e-05 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:753 3.0s0 3:0c3f65fa:::smithi18128752-8:head [append 0~0] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:24:56.123238+0000", 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 11.437396655000001, 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.71861978299999996, 2024-04-26T01:25:07.628 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 753 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123238+0000", 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.629 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123238+0000", 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123239+0000", 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5000000000000002e-07 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123242+0000", 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.9459999999999998e-06 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.630 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123254+0000", 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.2629999999999999e-05 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.123262+0000", 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.9039999999999992e-06 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.238821+0000", 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.11555886999999999 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.631 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.239046+0000", 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000225293 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.239900+0000", 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00085328699999999999 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.242822+0000", 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0029227020000000001 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.632 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.841718+0000", 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.59889568100000001 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:24:56.841858+0000", 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013972 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:819 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~491520] snapc 1f=[1f,1e,1d,1b] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:05.811695+0000", 2024-04-26T01:25:07.633 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.7489398089999999, 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.696380793, 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 819 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.811695+0000", 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.634 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.811695+0000", 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.811697+0000", 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6230000000000001e-06 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.812649+0000", 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00095168499999999997 2024-04-26T01:25:07.635 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.812679+0000", 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0162999999999999e-05 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.812694+0000", 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.4707e-05 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.812873+0000", 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00017877299999999999 2024-04-26T01:25:07.636 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.813175+0000", 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00030245199999999998 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.815304+0000", 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.002128567 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.817436+0000", 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0021321360000000002 2024-04-26T01:25:07.637 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.507929+0000", 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.69049305900000002 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.508076+0000", 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014662799999999999 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.638 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:820 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~802816] snapc 1f=[1f,1e,1d,1b] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:05.821771+0000", 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.738863708, 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.68694021900000002, 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 820 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.639 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.821771+0000", 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.821771+0000", 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.821774+0000", 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.6000000000000001e-06 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.640 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.822794+0000", 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001020698 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.822826+0000", 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.1542999999999999e-05 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.822841+0000", 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5129e-05 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.641 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.822946+0000", 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00010524200000000001 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.823129+0000", 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00018253400000000001 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.824721+0000", 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.001592605 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.642 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.826612+0000", 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0018903909999999999 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.508453+0000", 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.68184144099999999 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.508711+0000", 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00025803600000000001 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.643 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:821 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~720896] snapc 1f=[1f,1e,1d,1b] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:05.830821+0000", 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.7298137730000001, 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.67849151799999996, 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 821 2024-04-26T01:25:07.644 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.830821+0000", 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.830821+0000", 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.830823+0000", 2024-04-26T01:25:07.645 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 2.5170000000000002e-06 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831696+0000", 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000872279 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831727+0000", 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 3.0953e-05 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831742+0000", 2024-04-26T01:25:07.646 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.5469999999999999e-05 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.932000+0000", 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100258023 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.932195+0000", 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000195117 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.933815+0000", 2024-04-26T01:25:07.647 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.0016196019999999999 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.036465+0000", 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.102649794 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.509069+0000", 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.47260471399999998 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.509312+0000", 2024-04-26T01:25:07.648 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00024304900000000001 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:822 3.0s0 3:01bb5e34:::smithi18128752-45:head [append 0~32768] snapc 1f=[1f,1e,1d,1b] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:05.831773+0000", 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.7288612780000001, 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.70080558100000001, 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.649 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 822 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831773+0000", 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831773+0000", 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.650 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831775+0000", 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.35e-06 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831816+0000", 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.1288999999999997e-05 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831863+0000", 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 4.6589999999999999e-05 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.651 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831874+0000", 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.1012999999999999e-05 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.935095+0000", 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.103220987 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.935182+0000", 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.7917000000000005e-05 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.652 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.935821+0000", 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00063854000000000005 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.036619+0000", 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.100798255 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.532445+0000", 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49582549199999998 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.653 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.532579+0000", 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000134148 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:823 3.0s0 3:01bb5e34:::smithi18128752-45:head [setxattr _header (57) in=7b] snapc 1f=[1f,1e,1d,1b] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:05.831896+0000", 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.728738997, 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.80067409099999998, 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.654 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 823 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831896+0000", 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831896+0000", 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.655 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831897+0000", 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.7099999999999999e-06 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831905+0000", 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.5939999999999996e-06 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.656 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831923+0000", 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.8303000000000001e-05 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831937+0000", 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.3550000000000001e-05 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.138398+0000", 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30646103200000002 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.657 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.138594+0000", 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00019634700000000001 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_started", 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.139077+0000", 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00048274800000000002 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "sub_op_committed", 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.139822+0000", 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00074543400000000001 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "commit_sent", 2024-04-26T01:25:07.658 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.632429+0000", 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49260667800000002 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.632570+0000", 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00014069500000000001 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: "description": "osd_op(client.4589.0:824 3.0s0 3:01bb5e34:::smithi18128752-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: "initiated_at": "2024-04-26T01:25:05.831951+0000", 2024-04-26T01:25:07.659 INFO:teuthology.orchestra.run.smithi071.stdout: "age": 1.7286834679999998, 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0040148149999999, 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "continuous": false, 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "type_data": { 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "flag_point": "started", 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "client_info": { 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "client": "client.4589", 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "tid": 824 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "events": [ 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "initiated", 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831951+0000", 2024-04-26T01:25:07.660 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "header_read", 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831951+0000", 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "throttled", 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831952+0000", 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 8.6700000000000002e-07 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "all_read", 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831958+0000", 2024-04-26T01:25:07.661 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 6.1970000000000001e-06 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "dispatched", 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831969+0000", 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 1.0560000000000001e-05 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "queued_for_pg", 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:05.831977+0000", 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 7.8630000000000008e-06 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.662 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.139401+0000", 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.30742395 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.139538+0000", 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.00013767200000000001 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "reached_pg", 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.633723+0000", 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.49418506200000001 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.663 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "started", 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.633871+0000", 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.000147909 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: { 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: "event": "done", 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: "time": "2024-04-26T01:25:06.835966+0000", 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: "duration": 0.202094735 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-26T01:25:07.664 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-26T01:25:07.666 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:25:07.666 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:25:07.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:25:07.667 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:07.668+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:07.669 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:25:07.669 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:25:07.696 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:25:07.696 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:25:07.729 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:25:07.729 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:25:07.762 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:25:07.762 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:25:07.767 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:07.766+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:07.784 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:25:07.784 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:25:07.802 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:07.802 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:07.807 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:25:07.808 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:25:07.808 INFO:tasks.rados.rados.0.smithi181.stdout:334: done (3 left) 2024-04-26T01:25:07.808 INFO:tasks.rados.rados.0.smithi181.stdout:335: read oid 35 snap -1 2024-04-26T01:25:07.808 INFO:tasks.rados.rados.0.smithi181.stdout:335: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:336: read oid 23 snap -1 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:336: expect deleted 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:337: copy_from oid 7 from oid 21 current snap is 31 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: finishing write tid 6 to smithi18128752-45 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: oid 45 updating version 0 to 146 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: oid 45 updating version 146 to 147 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: oid 45 updating version 147 to 148 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: oid 45 version 148 is already newer than 144 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: oid 45 version 148 is already newer than 145 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 148 (ObjNum 121 snap 30 seq_num 121) dirty exists 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: left oid 45 (ObjNum 121 snap 30 seq_num 121) 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:329: done (5 left) 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:338: delete oid 31 current snap is 31 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:332: finishing write tid 1 to smithi18128752-43 2024-04-26T01:25:07.809 INFO:tasks.rados.rados.0.smithi181.stdout:333: expect (ObjNum 81 snap 13 seq_num 81) 2024-04-26T01:25:07.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:25:07.830 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:25:07.830 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:25:07.867 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:07.871+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:07.901 INFO:tasks.rados.rados.0.smithi181.stdout:332: finishing write tid 2 to smithi18128752-43 2024-04-26T01:25:07.901 INFO:tasks.rados.rados.0.smithi181.stdout:332: finishing write tid 3 to smithi18128752-43 2024-04-26T01:25:07.901 INFO:tasks.rados.rados.0.smithi181.stdout:332: finishing write tid 4 to smithi18128752-43 2024-04-26T01:25:07.901 INFO:tasks.rados.rados.0.smithi181.stdout:332: finishing write tid 5 to smithi18128752-43 2024-04-26T01:25:07.901 INFO:tasks.rados.rados.0.smithi181.stdout:332: finishing write tid 6 to smithi18128752-43 2024-04-26T01:25:07.939 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:25:07.940 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:25:07.949 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:07.949 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:07.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:25:07.968 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:07.971+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:08.069 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:08.071+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:08.070 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:08.070 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:08.081 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:08.081 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:08.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:25:08.104 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:08.104 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:08.137 INFO:tasks.rados.rados.0.smithi181.stdout:332: finishing write tid 7 to smithi18128752-43 2024-04-26T01:25:08.137 INFO:tasks.rados.rados.0.smithi181.stdout:332: oid 43 updating version 0 to 122 2024-04-26T01:25:08.137 INFO:tasks.rados.rados.0.smithi181.stdout:332: oid 43 updating version 122 to 123 2024-04-26T01:25:08.137 INFO:tasks.rados.rados.0.smithi181.stdout:332: oid 43 updating version 123 to 124 2024-04-26T01:25:08.138 INFO:tasks.rados.rados.0.smithi181.stdout:332: oid 43 updating version 124 to 125 2024-04-26T01:25:08.138 INFO:tasks.rados.rados.0.smithi181.stdout:332: oid 43 version 125 is already newer than 121 2024-04-26T01:25:08.138 INFO:tasks.rados.rados.0.smithi181.stdout:332: oid 43 version 125 is already newer than 120 2024-04-26T01:25:08.138 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 125 (ObjNum 122 snap 31 seq_num 122) dirty exists 2024-04-26T01:25:08.138 INFO:tasks.rados.rados.0.smithi181.stdout:332: left oid 43 (ObjNum 122 snap 31 seq_num 122) 2024-04-26T01:25:08.170 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:08.175+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:08.221 INFO:tasks.rados.rados.0.smithi181.stdout:337: finishing copy_from racing read to smithi18128752-7 2024-04-26T01:25:08.223 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:08.223 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:08.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:25:08.284 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:08.288+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:08.345 INFO:tasks.rados.rados.0.smithi181.stdout:332: done (5 left) 2024-04-26T01:25:08.345 INFO:tasks.rados.rados.0.smithi181.stdout:333: done (4 left) 2024-04-26T01:25:08.345 INFO:tasks.rados.rados.0.smithi181.stdout:336: done (3 left) 2024-04-26T01:25:08.345 INFO:tasks.rados.rados.0.smithi181.stdout:338: done (2 left) 2024-04-26T01:25:08.345 INFO:tasks.rados.rados.0.smithi181.stdout:339: snap_create 2024-04-26T01:25:08.373 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:08.374+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:08.407 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:08.407 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:08.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:25:08.427 INFO:tasks.rados.rados.0.smithi181.stdout:337: finishing copy_from to smithi18128752-7 2024-04-26T01:25:08.427 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 7 v 59 (ObjNum 92 snap 17 seq_num 92) dirty exists 2024-04-26T01:25:08.474 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:08.476+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:08.524 INFO:tasks.rados.rados.0.smithi181.stdout:335: expect (ObjNum 34 snap 0 seq_num 34) 2024-04-26T01:25:08.544 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:08.544 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:08.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:25:08.600 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:08.604+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:08.676 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:08.679+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:08.723 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:08.723 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:08.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:25:08.777 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:08.779+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:08.790 INFO:tasks.rados.rados.0.smithi181.stdout:335: done (2 left) 2024-04-26T01:25:08.790 INFO:tasks.rados.rados.0.smithi181.stdout:337: done (1 left) 2024-04-26T01:25:08.790 INFO:tasks.rados.rados.0.smithi181.stdout:339: done (0 left) 2024-04-26T01:25:08.790 INFO:tasks.rados.rados.0.smithi181.stdout:340: delete oid 19 current snap is 32 2024-04-26T01:25:08.823 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-26T01:25:08.860 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:08.860 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:08.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:25:08.878 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:08.879+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:08.978 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:08.983+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:08.996 INFO:tasks.rados.rados.0.smithi181.stdout:340: done (0 left) 2024-04-26T01:25:08.996 INFO:tasks.rados.rados.0.smithi181.stdout:341: delete oid 8 current snap is 32 2024-04-26T01:25:09.003 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:09.003 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:09.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:25:09.083 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:09.084+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:09.205 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:09.205 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:09.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:25:09.206 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:09.206+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:09.307 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:09.311+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:09.373 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:09.373 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:09.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:25:09.407 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:09.411+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:09.502 INFO:tasks.rados.rados.0.smithi181.stdout:341: done (0 left) 2024-04-26T01:25:09.502 INFO:tasks.rados.rados.0.smithi181.stdout:342: delete oid 49 current snap is 32 2024-04-26T01:25:09.508 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:09.511+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:09.511 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:09.512 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:09.526 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:25:09.609 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:09.611+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:09.663 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:09.663 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:09.678 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:25:09.708 INFO:tasks.rados.rados.0.smithi181.stdout:342: done (0 left) 2024-04-26T01:25:09.708 INFO:tasks.rados.rados.0.smithi181.stdout:343: copy_from oid 33 from oid 9 current snap is 32 2024-04-26T01:25:09.708 INFO:tasks.rados.rados.0.smithi181.stdout:344: read oid 27 snap -1 2024-04-26T01:25:09.708 INFO:tasks.rados.rados.0.smithi181.stdout:344: expect deleted 2024-04-26T01:25:09.709 INFO:tasks.rados.rados.0.smithi181.stdout:345: rollback oid 42 current snap is 32 2024-04-26T01:25:09.709 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 42 to 29 2024-04-26T01:25:09.709 INFO:tasks.rados.rados.0.smithi181.stdout:346: read oid 29 snap -1 2024-04-26T01:25:09.709 INFO:tasks.rados.rados.0.smithi181.stdout:346: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:25:09.709 INFO:tasks.rados.rados.0.smithi181.stdout:347: delete oid 19 current snap is 32 2024-04-26T01:25:09.710 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:09.715+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:09.804 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:09.804 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:09.811 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:09.815+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:09.818 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:25:09.913 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:09.914+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:09.946 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:09.946 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:09.960 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:25:09.985 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is set 2024-04-26T01:25:10.050 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:10.055+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:10.089 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:10.089 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:10.104 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:25:10.115 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:10.114+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:10.117 INFO:tasks.rados.rados.0.smithi181.stdout:345: finishing rollback tid 0 to smithi18128752-42 2024-04-26T01:25:10.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:25:10.216 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:10.218+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:10.274 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:10.274 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:10.290 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:25:10.294 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:25:10.294 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:25:10.316 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:25:10.317 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:25:10.317 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:10.319+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:10.319 INFO:tasks.rados.rados.0.smithi181.stdout:345: finishing rollback tid 1 to smithi18128752-42 2024-04-26T01:25:10.319 INFO:tasks.rados.rados.0.smithi181.stdout:346: expect (ObjNum 28 snap 0 seq_num 28) 2024-04-26T01:25:10.339 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:25:10.339 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:25:10.360 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:25:10.360 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:25:10.377 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:25:10.378 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:25:10.398 INFO:tasks.rados.rados.0.smithi181.stdout:345: finishing rollback tid 2 to smithi18128752-42 2024-04-26T01:25:10.398 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 42 v 158 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-04-26T01:25:10.412 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:10.412 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:25:10.412 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:123 3.ds0 3:b6ed1d05:::smithi18128752-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 64.920427865999997, 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.401407085, 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.413 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 123 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:25:10.414 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493545+0000", 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5299999999999999e-07 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493546+0000", 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7039999999999999e-06 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.415 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493552+0000", 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5119999999999999e-06 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493556+0000", 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2679999999999996e-06 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697434+0000", 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20387831000000001 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697559+0000", 2024-04-26T01:25:10.416 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012467700000000001 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087644+0000", 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39008522000000001 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.417 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087733+0000", 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9191999999999996e-05 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894951+0000", 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80721774899999998 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.418 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:138 3.1s0 3:8830b48d:::smithi18128752-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 64.634550060999999, 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71477871400000004, 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.419 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 138 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.420 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779423+0000", 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.47e-07 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779429+0000", 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6400000000000002e-06 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779443+0000", 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4392999999999999e-05 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.421 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779456+0000", 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2643000000000001e-05 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784323+0000", 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048675369999999999 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784379+0000", 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6314000000000003e-05 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.422 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187338+0000", 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40295840700000002 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187417+0000", 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8898000000000003e-05 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.494201+0000", 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30678393500000001 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.423 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:140 3.ds0 3:b9916085:::smithi18128752-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 64.311095163999994, 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.69088028800000001, 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 140 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.424 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102877+0000", 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102878+0000", 2024-04-26T01:25:10.425 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2399999999999997e-07 2024-04-26T01:25:10.426 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.426 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.426 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.426 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102881+0000", 2024-04-26T01:25:10.426 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1489999999999998e-06 2024-04-26T01:25:10.426 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.426 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102885+0000", 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.1300000000000003e-06 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102888+0000", 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.242e-06 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.290947+0000", 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.18805879 2024-04-26T01:25:10.427 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291087+0000", 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013990199999999999 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291629+0000", 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00054169199999999998 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.492101+0000", 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20047221400000001 2024-04-26T01:25:10.428 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.793519+0000", 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30141857300000002 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.793757+0000", 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000237772 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.429 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:141 3.ds0 3:b9916085:::smithi18128752-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 64.311078992999995, 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.99325914900000001, 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 141 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.430 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.431 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102896+0000", 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3209999999999999e-06 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102900+0000", 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9369999999999997e-06 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102903+0000", 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3809999999999999e-06 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.432 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291929+0000", 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.189026048 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.292027+0000", 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8172999999999997e-05 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794279+0000", 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50225164799999999 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.433 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794416+0000", 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013710400000000001 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.096152+0000", 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30173626199999998 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:325 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:10.434 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 52.052646422999999, 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70936785199999997, 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 325 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:25:10.435 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.436 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361328+0000", 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3580000000000001e-06 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361338+0000", 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.628e-06 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361360+0000", 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2506000000000001e-05 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361374+0000", 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4072e-05 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454039+0000", 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.092664721000000005 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454298+0000", 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025867499999999999 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.455951+0000", 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016533489999999999 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462924+0000", 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069735650000000001 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070553+0000", 2024-04-26T01:25:10.439 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60762835599999998 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070693+0000", 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000140622 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:326 3.7s0 3:e977e128:::smithi18128752-21:head [rollback 4] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 52.052577857000003, 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70968584099999998, 2024-04-26T01:25:10.440 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 326 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361395+0000", 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.06e-07 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361398+0000", 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1379999999999999e-06 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.442 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361408+0000", 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.046e-05 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361414+0000", 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8459999999999996e-06 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458365+0000", 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096950878000000004 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458663+0000", 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000297788 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.460203+0000", 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015404030000000001 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.463428+0000", 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032243850000000002 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070927+0000", 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.607499821 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071080+0000", 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000152516 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:327 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:10.445 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 52.052550031000003, 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71007962499999999, 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 327 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.446 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.447 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361425+0000", 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2740000000000002e-06 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361431+0000", 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4280000000000001e-06 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361436+0000", 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.994e-06 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.448 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461250+0000", 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099813814000000001 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461584+0000", 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000334521 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462509+0000", 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00092412699999999996 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.449 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.565364+0000", 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102855424 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071318+0000", 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50595366500000005 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071502+0000", 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000183912 2024-04-26T01:25:10.450 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:328 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 52.052529626999998, 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81248642000000004, 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.451 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 328 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361443+0000", 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6199999999999998e-07 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361447+0000", 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.262e-06 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361454+0000", 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8739999999999997e-06 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.453 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361459+0000", 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8040000000000001e-06 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563609+0000", 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20214983 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563788+0000", 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017973199999999999 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.173929+0000", 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.610140456 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:329 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:10.455 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 52.052507030999998, 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81314448500000003, 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 329 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:10.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2399999999999999e-07 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361469+0000", 2024-04-26T01:25:10.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2820000000000001e-06 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361475+0000", 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3509999999999998e-06 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361480+0000", 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6959999999999996e-06 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.458 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564389+0000", 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202908808 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564593+0000", 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000204255 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.174609+0000", 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.61001666899999996 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.459 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:330 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 52.052486520000002, 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81407258199999999, 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 330 2024-04-26T01:25:10.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361486+0000", 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0999999999999999e-07 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361489+0000", 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7240000000000001e-06 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361495+0000", 2024-04-26T01:25:10.462 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2740000000000004e-06 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361500+0000", 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9400000000000001e-06 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.866935+0000", 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50543468599999997 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.867233+0000", 2024-04-26T01:25:10.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029877499999999998 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.175558+0000", 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.308324773 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:434 3.7s0 3:e9e07e85:::smithi18128752-17:head [copy-from ver 29 in=54b] snapc d=[d,c,8,4] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 41.635319840999998, 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1051031140000001, 2024-04-26T01:25:10.464 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 434 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778655+0000", 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8770000000000002e-06 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778667+0000", 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1888999999999999e-05 2024-04-26T01:25:10.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778697+0000", 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0205999999999998e-05 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778714+0000", 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6467999999999999e-05 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878928+0000", 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100213993 2024-04-26T01:25:10.467 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878940+0000", 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2724999999999999e-05 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882181+0000", 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032411419999999998 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882351+0000", 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169371 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.678907+0000", 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.79655591199999998 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.882986+0000", 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.204079063 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.469 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883081+0000", 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5285000000000007e-05 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883755+0000", 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00067418299999999997 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:435 3.7s0 3:e9e07e85:::smithi18128752-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 41.635224184000002, 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.003729404, 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 435 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778749+0000", 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.223e-06 2024-04-26T01:25:10.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778755+0000", 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8279999999999999e-06 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778773+0000", 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8289000000000001e-05 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878906+0000", 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10013274699999999 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879035+0000", 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012864199999999999 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.474 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879049+0000", 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4137e-05 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882650+0000", 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036017390000000001 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882716+0000", 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5129000000000006e-05 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782226+0000", 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.89951035899999998 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782341+0000", 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000115492 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782477+0000", 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000135819 2024-04-26T01:25:10.476 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:477 3.bs0 3:d65fea44:::smithi18128752-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 37.521392544999998, 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.91386205600000003, 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.477 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 477 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892580+0000", 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7999999999999996e-07 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892582+0000", 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9630000000000001e-06 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892589+0000", 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7959999999999998e-06 2024-04-26T01:25:10.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892593+0000", 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2509999999999996e-06 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992304+0000", 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099710909 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992356+0000", 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2021000000000002e-05 2024-04-26T01:25:10.480 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402928+0000", 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.41057238600000001 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.403133+0000", 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020520099999999999 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.806441+0000", 2024-04-26T01:25:10.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.403308049 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:537 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~704512] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.586804397000002, 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77667111300000002, 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.482 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 537 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.483 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827172+0000", 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9639999999999997e-06 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.828324+0000", 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011519550000000001 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.484 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.828352+0000", 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8112000000000001e-05 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928473+0000", 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100121081 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.994151+0000", 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.065677997000000002 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.994232+0000", 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1688999999999997e-05 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.995096+0000", 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00086366299999999997 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997758+0000", 2024-04-26T01:25:10.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0026614830000000001 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.603693+0000", 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60593560000000002 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.603839+0000", 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014556899999999999 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.487 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:538 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~688128] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.928521+0000", 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.485451109, 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.67561050700000003, 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.488 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 538 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928521+0000", 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.489 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928521+0000", 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928523+0000", 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6449999999999999e-06 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929086+0000", 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00056332400000000003 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929102+0000", 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5951999999999999e-05 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929110+0000", 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.2730000000000002e-06 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.996156+0000", 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.067045503000000006 2024-04-26T01:25:10.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.996259+0000", 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000103255 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997239+0000", 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00098057200000000008 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999670+0000", 2024-04-26T01:25:10.492 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0024309980000000002 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.604023+0000", 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60435304599999995 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.604131+0000", 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000107939 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.493 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:539 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~81920] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.484851201000001, 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77547855799999998, 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.494 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 539 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:10.495 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5400000000000001e-07 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929175+0000", 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4033000000000003e-05 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929183+0000", 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7130000000000005e-06 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929187+0000", 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2719999999999999e-06 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997790+0000", 2024-04-26T01:25:10.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.068602814999999998 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997917+0000", 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000126783 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.498 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998505+0000", 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00058842300000000003 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999674+0000", 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011685179999999999 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704445+0000", 2024-04-26T01:25:10.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.704771335 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704599+0000", 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000154012 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:540 3.1s0 3:865c3313:::smithi18128752-37:head [setxattr _header (57) in=7b] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.484778157000001, 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77584385600000005, 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 540 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9200000000000002e-07 2024-04-26T01:25:10.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929197+0000", 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.661e-06 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929201+0000", 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.194e-06 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929204+0000", 2024-04-26T01:25:10.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4020000000000002e-06 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998891+0000", 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.069686039000000005 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998997+0000", 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010671 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999475+0000", 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00047769400000000002 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999988+0000", 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051352600000000004 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704829+0000", 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70484043500000004 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.705038+0000", 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000208803 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:541 3.1s0 3:865c3313:::smithi18128752-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 31.484762887999999, 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0548109999999999, 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.506 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 541 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.507 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9999999999999999e-07 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929211+0000", 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.517e-06 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929217+0000", 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0410000000000003e-06 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929221+0000", 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.659e-06 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.028709+0000", 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099488691000000004 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.028788+0000", 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8268000000000003e-05 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.707449+0000", 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.67866140799999997 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.707686+0000", 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023745600000000001 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.984020+0000", 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.27633365999999998 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:760 3.1s0 3:8830b48d:::smithi18128752-46:head [setxattr _header (57) in=7b] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.225249+0000", 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.188722612999999, 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71216155800000003, 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.511 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 760 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225249+0000", 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.512 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225249+0000", 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225251+0000", 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.237e-06 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225257+0000", 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5370000000000003e-06 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225270+0000", 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3305999999999999e-05 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225279+0000", 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5969999999999992e-06 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236485+0000", 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011206317 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236734+0000", 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000248206 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237890+0000", 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001156575 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.238966+0000", 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010758689999999999 2024-04-26T01:25:10.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.937219+0000", 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.69825322400000001 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.937411+0000", 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019169 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:761 3.1s0 3:8830b48d:::smithi18128752-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.225291+0000", 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.188681076, 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71929514500000002, 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 761 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225291+0000", 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225291+0000", 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225292+0000", 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6700000000000003e-07 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225295+0000", 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4759999999999998e-06 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225306+0000", 2024-04-26T01:25:10.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0883999999999999e-05 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225314+0000", 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1799999999999996e-06 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.330700+0000", 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10538581800000001 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.330964+0000", 2024-04-26T01:25:10.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026375200000000002 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.938144+0000", 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60718067099999995 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.938316+0000", 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017196600000000001 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:10.521 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.944586+0000", 2024-04-26T01:25:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0062696310000000003 2024-04-26T01:25:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.522 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.522 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:10.523 INFO:tasks.rados.rados.0.smithi181.stdout:343: finishing copy_from racing read to smithi18128752-33 2024-04-26T01:25:10.523 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:25:10.524 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:25:10.525 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:25:10.525 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:25:10.525 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:25:10.526 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:25:10.526 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:25:10.526 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:25:10.527 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:25:10.529 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:25:10.530 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:25:10.530 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:10.528+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:10.545 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:25:10.546 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:25:10.570 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:25:10.570 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:25:10.591 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:25:10.591 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:344: done (4 left) 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:345: done (3 left) 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:346: done (2 left) 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:347: done (1 left) 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:348: rollback oid 17 current snap is 32 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 17 to 27 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:349: rollback oid 34 current snap is 32 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 34 to 28 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 32 2024-04-26T01:25:10.600 INFO:tasks.rados.rados.0.smithi181.stdout:350: seq_num 128 ranges {3457024=606208,4063232=507904,4571136=573440,5144576=475136} 2024-04-26T01:25:10.618 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:25:10.618 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:25:10.624 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:10.627+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:10.641 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:10.641 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:10.662 INFO:tasks.rados.rados.0.smithi181.stdout:350: writing smithi18128752-50 from 3457024 to 4063232 tid 1 2024-04-26T01:25:10.665 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:10.665 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:10.667 INFO:tasks.rados.rados.0.smithi181.stdout:350: writing smithi18128752-50 from 4063232 to 4571136 tid 2 2024-04-26T01:25:10.673 INFO:tasks.rados.rados.0.smithi181.stdout:350: writing smithi18128752-50 from 4571136 to 5144576 tid 3 2024-04-26T01:25:10.678 INFO:tasks.rados.rados.0.smithi181.stdout:350: writing smithi18128752-50 from 5144576 to 5619712 tid 4 2024-04-26T01:25:10.678 INFO:tasks.rados.rados.0.smithi181.stdout:351: read oid 31 snap -1 2024-04-26T01:25:10.679 INFO:tasks.rados.rados.0.smithi181.stdout:351: expect deleted 2024-04-26T01:25:10.679 INFO:tasks.rados.rados.0.smithi181.stdout:352: rollback oid 43 current snap is 32 2024-04-26T01:25:10.679 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 43 to 27 2024-04-26T01:25:10.679 INFO:tasks.rados.rados.0.smithi181.stdout:343: finishing copy_from to smithi18128752-33 2024-04-26T01:25:10.679 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 45 (ObjNum 82 snap 13 seq_num 82) dirty exists 2024-04-26T01:25:10.679 INFO:tasks.rados.rados.0.smithi181.stdout:343: done (5 left) 2024-04-26T01:25:10.679 INFO:tasks.rados.rados.0.smithi181.stdout:353: snap_create 2024-04-26T01:25:10.687 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s4 98/30 ECSubWrite(tid=323, reqid=client.4589.0:866, at_version=98'106, trim_to=0'0, roll_forward_to=94'103))", 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.610052+0000", 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.083610478000000002, 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.083637813000000005, 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.610052+0000", 2024-04-26T01:25:10.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.610052+0000", 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.610053+0000", 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.071e-06 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.610075+0000", 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1861999999999998e-05 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.610089+0000", 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3625000000000001e-05 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.690 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.610100+0000", 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1148000000000001e-05 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s4 98/30 ECSubWrite(tid=324, reqid=client.4589.0:867, at_version=98'107, trim_to=0'0, roll_forward_to=94'103))", 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.611647+0000", 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.082016088000000001, 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.082082550000000004, 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:10.691 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.611647+0000", 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.611647+0000", 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.611648+0000", 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6899999999999996e-07 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.611663+0000", 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5843000000000001e-05 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.611685+0000", 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1447e-05 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.611695+0000", 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0108e-05 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 98/90 ECSubWrite(tid=146, reqid=client.4589.0:869, at_version=98'92, trim_to=0'0, roll_forward_to=94'90))", 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.672862+0000", 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.020801147999999998, 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.020913798000000001, 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.694 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.672862+0000", 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.672862+0000", 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.672862+0000", 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.66e-07 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.673561+0000", 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00069929699999999998 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.673583+0000", 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.143e-05 2024-04-26T01:25:10.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.673597+0000", 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4003e-05 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677831+0000", 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0042343199999999998 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677849+0000", 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8025000000000001e-05 2024-04-26T01:25:10.697 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs1 98/90 ECSubWrite(tid=147, reqid=client.4589.0:870, at_version=98'93, trim_to=0'0, roll_forward_to=94'90))", 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.677281+0000", 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.016382166, 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.016543634000000002, 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677281+0000", 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677281+0000", 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677282+0000", 2024-04-26T01:25:10.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3430000000000001e-06 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677655+0000", 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00037259400000000002 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677683+0000", 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8765e-05 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.677699+0000", 2024-04-26T01:25:10.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5196e-05 2024-04-26T01:25:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T01:25:10.701 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 4 2024-04-26T01:25:10.701 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:10.703 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:25:10.726 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:10.728+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:10.827 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:10.826+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:10.830 INFO:tasks.rados.rados.0.smithi181.stdout:348: finishing rollback tid 0 to smithi18128752-17 2024-04-26T01:25:10.830 INFO:tasks.rados.rados.0.smithi181.stdout:348: finishing rollback tid 1 to smithi18128752-17 2024-04-26T01:25:10.831 INFO:tasks.rados.rados.0.smithi181.stdout:348: finishing rollback tid 2 to smithi18128752-17 2024-04-26T01:25:10.831 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 111 (ObjNum 102 snap 19 seq_num 102) dirty exists 2024-04-26T01:25:10.874 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:10.874 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:10.874 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:25:10.875 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:25:10.875 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:10.889 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:25:10.928 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:10.926+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:10.987 INFO:tasks.rados.rados.0.smithi181.stdout:350: finishing write tid 1 to smithi18128752-50 2024-04-26T01:25:11.003 INFO:tasks.rados.rados.0.smithi181.stdout:348: done (5 left) 2024-04-26T01:25:11.003 INFO:tasks.rados.rados.0.smithi181.stdout:351: done (4 left) 2024-04-26T01:25:11.003 INFO:tasks.rados.rados.0.smithi181.stdout:353: done (3 left) 2024-04-26T01:25:11.003 INFO:tasks.rados.rados.0.smithi181.stdout:354: rollback oid 40 current snap is 33 2024-04-26T01:25:11.004 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 40 to 29 2024-04-26T01:25:11.004 INFO:tasks.rados.rados.0.smithi181.stdout:355: rmattr oid 2 current snap is 33 2024-04-26T01:25:11.004 INFO:tasks.rados.rados.0.smithi181.stdout:355: done (4 left) 2024-04-26T01:25:11.004 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 48 current snap is 33 2024-04-26T01:25:11.004 INFO:tasks.rados.rados.0.smithi181.stdout:356: seq_num 129 ranges {1638400=622592,2260992=655360,2916352=688128,3604480=212992} 2024-04-26T01:25:11.017 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:11.017 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:25:11.017 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:25:11.017 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:11.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.017 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:187 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~0] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 62.846358815999999, 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51780501899999998, 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 187 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.018 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173374+0000", 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173375+0000", 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1699999999999999e-07 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173377+0000", 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.939e-06 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173381+0000", 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2050000000000004e-06 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.020 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.173384+0000", 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2760000000000001e-06 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264258+0000", 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.090873535000000005 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.021 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.264532+0000", 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027448400000000002 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.265351+0000", 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00081914900000000003 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.022 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.466871+0000", 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20151915000000001 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.690961+0000", 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22409068500000001 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:08.691180+0000", 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021827899999999999 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.023 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:271 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 58.846717353000003, 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53167738200000003, 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 271 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.024 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173016+0000", 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173017+0000", 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1100000000000005e-07 2024-04-26T01:25:11.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173022+0000", 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4990000000000002e-06 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173034+0000", 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1667e-05 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173044+0000", 2024-04-26T01:25:11.026 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9070000000000002e-06 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398177+0000", 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225132687 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.398388+0000", 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000211325 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.027 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.704693+0000", 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30630548600000002 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:272 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~458752,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 58.846675673999997, 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.53273070499999997, 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.028 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 272 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173058+0000", 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.9200000000000001e-07 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173079+0000", 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1137000000000001e-05 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173093+0000", 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3198e-05 2024-04-26T01:25:11.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.173102+0000", 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1859999999999998e-06 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401582+0000", 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22848043000000001 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.401813+0000", 2024-04-26T01:25:11.031 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023127900000000001 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:12.705788+0000", 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30397488299999997 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:611 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~540672] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.812212+0000", 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.207521756999999, 2024-04-26T01:25:11.032 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50597818100000003, 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 611 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.812212+0000", 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.812212+0000", 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.812214+0000", 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6769999999999999e-06 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.813057+0000", 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00084302800000000001 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.813088+0000", 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0865999999999999e-05 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.813102+0000", 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4239999999999999e-05 2024-04-26T01:25:11.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913298+0000", 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100195669 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913306+0000", 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6909999999999997e-06 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.914357+0000", 2024-04-26T01:25:11.036 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00105167 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.914510+0000", 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015237000000000001 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.916163+0000", 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001653259 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921172+0000", 2024-04-26T01:25:11.037 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0050092299999999999 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.317916+0000", 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.396743868 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.318190+0000", 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027361300000000002 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.038 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:612 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~507904] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.816201+0000", 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.203532219, 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50253841700000002, 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 612 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.039 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816201+0000", 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816201+0000", 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816206+0000", 2024-04-26T01:25:11.040 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3460000000000001e-06 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816745+0000", 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00053856399999999995 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816773+0000", 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8017e-05 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.816787+0000", 2024-04-26T01:25:11.041 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3854e-05 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913509+0000", 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096722404999999997 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.913519+0000", 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.6800000000000005e-06 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.042 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917532+0000", 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0040130610000000001 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.917712+0000", 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000180156 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.919317+0000", 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001604891 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.043 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921251+0000", 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00193424 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.318533+0000", 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39728167399999997 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.318739+0000", 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000206529 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.044 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:613 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~622592] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.821128+0000", 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.198605251, 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59873126099999996, 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 613 2024-04-26T01:25:11.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.821128+0000", 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.821128+0000", 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.821131+0000", 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5239999999999999e-06 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822321+0000", 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001190485 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822353+0000", 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2147000000000002e-05 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.047 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822369+0000", 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5593000000000001e-05 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921300+0000", 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098931370000000005 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.921469+0000", 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016842499999999999 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.048 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.922991+0000", 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001522036 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.927109+0000", 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0041182099999999998 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.419729+0000", 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.49262054799999999 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.419859+0000", 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000129923 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:614 3.4s0 3:2e10a639:::smithi18128752-28:head [append 0~32768] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822397+0000", 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.197335898999999, 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59784716400000004, 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.050 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 614 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822397+0000", 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822397+0000", 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822399+0000", 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2529999999999999e-06 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822459+0000", 2024-04-26T01:25:11.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0736000000000003e-05 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822478+0000", 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8193999999999999e-05 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822488+0000", 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.028e-05 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.924397+0000", 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101909182 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.924681+0000", 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028437299999999998 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.926252+0000", 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015702299999999999 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.054 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929400+0000", 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031478809999999999 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420107+0000", 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.490707952 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420245+0000", 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000137083 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.055 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:615 3.4s0 3:2e10a639:::smithi18128752-28:head [setxattr _header (57) in=7b] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822505+0000", 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.197228460000002, 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59808836700000001, 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.056 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 615 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822505+0000", 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822505+0000", 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822506+0000", 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8000000000000004e-07 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822511+0000", 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8369999999999996e-06 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822524+0000", 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.312e-05 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822533+0000", 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.02e-06 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.927335+0000", 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.104802701 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.927618+0000", 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028260900000000002 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.928984+0000", 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001365851 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930534+0000", 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001550608 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420432+0000", 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.48989737799999999 2024-04-26T01:25:11.060 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420593+0000", 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016136300000000001 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:616 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822544+0000", 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.197189196, 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80089191999999998, 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.061 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 616 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822544+0000", 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822544+0000", 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822545+0000", 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5999999999999991e-07 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822550+0000", 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2299999999999999e-06 2024-04-26T01:25:11.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822564+0000", 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3499999999999999e-05 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822572+0000", 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4200000000000007e-06 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929502+0000", 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.106929992 2024-04-26T01:25:11.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929766+0000", 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026414899999999998 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420729+0000", 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.49096291800000003 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.420898+0000", 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016868300000000001 2024-04-26T01:25:11.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.623436+0000", 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20253806799999999 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:617 3.4s0 3:2be1e129:::smithi18128752-30:head [copy-from ver 59] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822584+0000", 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.197149020000001, 2024-04-26T01:25:11.066 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0342352619999999, 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 617 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822584+0000", 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822584+0000", 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822585+0000", 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3e-07 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822590+0000", 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.707e-06 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822602+0000", 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.243e-05 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822610+0000", 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7030000000000006e-06 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.929881+0000", 2024-04-26T01:25:11.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.107270828 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930159+0000", 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027794499999999998 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.647173+0000", 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71701393899999999 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653979+0000", 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0068063760000000003 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.856678+0000", 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20269896000000001 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.856820+0000", 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014164399999999999 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:618 3.4s0 3:2be1e129:::smithi18128752-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 27.197112579999999, 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.83091384000000001, 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 618 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.072 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822621+0000", 2024-04-26T01:25:11.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6700000000000003e-07 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822625+0000", 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3900000000000002e-06 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822637+0000", 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2033e-05 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.822645+0000", 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6559999999999992e-06 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930630+0000", 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.107985175 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.930789+0000", 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015969000000000001 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653189+0000", 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.72239973400000002 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653353+0000", 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016386600000000001 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.653535+0000", 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018152899999999999 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.076 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:718 3.4s0 3:2e10a639:::smithi18128752-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83)", 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:54.866892+0000", 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 16.152841618, 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.59924005700000005, 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 718 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.077 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.866892+0000", 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.866892+0000", 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.866892+0000", 2024-04-26T01:25:11.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2199999999999999e-07 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.866894+0000", 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9920000000000002e-06 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.866899+0000", 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0479999999999998e-06 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.866902+0000", 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.337e-06 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.868703+0000", 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0018002179999999999 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:54.868771+0000", 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8021999999999993e-05 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.264993+0000", 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.396222291 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.265100+0000", 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000106665 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.466132+0000", 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20103206200000001 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:740 3.4s0 3:2e3ebee3:::smithi18128752-33:head [append 0~0] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:55.966906+0000", 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.052827795000001, 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.55922566500000004, 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 740 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.082 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.966906+0000", 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.966906+0000", 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.966909+0000", 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5200000000000002e-06 2024-04-26T01:25:11.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.966924+0000", 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5104e-05 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.966963+0000", 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8525999999999997e-05 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.966983+0000", 2024-04-26T01:25:11.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0163999999999999e-05 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.067193+0000", 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10020973399999999 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.067297+0000", 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010438 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.068211+0000", 2024-04-26T01:25:11.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00091385399999999997 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.171391+0000", 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10318058400000001 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.525989+0000", 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.35459741 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.526131+0000", 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014238900000000001 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:741 3.4s0 3:2e3ebee3:::smithi18128752-33:head [rollback 18] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:55.967026+0000", 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.052707089, 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.66062653000000005, 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.087 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 741 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967026+0000", 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967026+0000", 2024-04-26T01:25:11.088 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967027+0000", 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.9399999999999993e-07 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967032+0000", 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6199999999999998e-06 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967043+0000", 2024-04-26T01:25:11.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1473e-05 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967054+0000", 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.092e-05 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.068796+0000", 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101741874 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.068890+0000", 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.4371999999999997e-05 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.069441+0000", 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000550845 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.171706+0000", 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102264197 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.627515+0000", 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.45580994200000002 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.627653+0000", 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000137293 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:742 3.4s0 3:2e3ebee3:::smithi18128752-33:head [append 0~0] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:55.967112+0000", 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.052621466, 2024-04-26T01:25:11.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.66115535400000003, 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 742 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967112+0000", 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967112+0000", 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967113+0000", 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9999999999999996e-07 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967118+0000", 2024-04-26T01:25:11.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0599999999999998e-06 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967134+0000", 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5724000000000001e-05 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:55.967143+0000", 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7059999999999999e-06 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.169843+0000", 2024-04-26T01:25:11.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20270004 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.170072+0000", 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000229146 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.170873+0000", 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00080050300000000004 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.276283+0000", 2024-04-26T01:25:11.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.105409871 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.627981+0000", 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.35169799600000001 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.628267+0000", 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028640799999999999 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:743 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~770048] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.013650+0000", 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.006083508, 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.61499042800000003, 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 743 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.013650+0000", 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.013650+0000", 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.013652+0000", 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6170000000000001e-06 2024-04-26T01:25:11.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.014628+0000", 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00097597799999999996 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.014665+0000", 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6755999999999998e-05 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.014685+0000", 2024-04-26T01:25:11.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0007e-05 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.171877+0000", 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.157192095 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.172029+0000", 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015198699999999999 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.173811+0000", 2024-04-26T01:25:11.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0017818389999999999 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.276294+0000", 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102482606 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.628519+0000", 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.35222571000000003 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.628640+0000", 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000120833 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:744 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~704512] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.017736+0000", 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 15.001997713, 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.61146661400000002, 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.103 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 744 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.017736+0000", 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.017736+0000", 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.104 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.017738+0000", 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6369999999999999e-06 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.018818+0000", 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001080009 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.018845+0000", 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6299999999999999e-05 2024-04-26T01:25:11.105 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.018859+0000", 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4137e-05 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.176202+0000", 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.157343597 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.176370+0000", 2024-04-26T01:25:11.106 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016801300000000001 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.177849+0000", 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014785270000000001 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.276295+0000", 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.098446599999999995 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.629003+0000", 2024-04-26T01:25:11.107 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.352707612 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.629202+0000", 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000199182 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:745 3.4s0 3:2be1e129:::smithi18128752-30:head [append 0~458752] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.021139+0000", 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.99859468, 2024-04-26T01:25:11.108 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60913616599999998, 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 745 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.021139+0000", 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.109 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.021139+0000", 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.021142+0000", 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6569999999999999e-06 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022207+0000", 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010647160000000001 2024-04-26T01:25:11.110 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022244+0000", 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7319e-05 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022270+0000", 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5746999999999999e-05 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.178867+0000", 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.15659735699999999 2024-04-26T01:25:11.111 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.179016+0000", 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014861 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.180230+0000", 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012143690000000001 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.276297+0000", 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096067026 2024-04-26T01:25:11.112 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.630079+0000", 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.35378188300000002 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.630275+0000", 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000195482 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.113 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:746 3.4s0 3:2be1e129:::smithi18128752-30:head [setxattr _header (57) in=7b] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.022314+0000", 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.997419795000001, 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60839292899999997, 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 746 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.114 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022314+0000", 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022314+0000", 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022316+0000", 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1730000000000002e-06 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.115 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022326+0000", 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0353999999999999e-05 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022346+0000", 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0156000000000001e-05 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022359+0000", 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2594e-05 2024-04-26T01:25:11.116 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.181193+0000", 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.15883382900000001 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.181340+0000", 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014729899999999999 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.182022+0000", 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00068165999999999995 2024-04-26T01:25:11.117 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.276299+0000", 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.094277929999999996 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.630535+0000", 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.35423567500000003 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.630706+0000", 2024-04-26T01:25:11.118 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017125899999999999 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:747 3.4s0 3:2be1e129:::smithi18128752-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.022385+0000", 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 14.997348182, 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0123909149999999, 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:11.119 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 747 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022385+0000", 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022385+0000", 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.120 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022387+0000", 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4670000000000001e-06 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022394+0000", 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.1470000000000004e-06 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022412+0000", 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8253e-05 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.121 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.022424+0000", 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2510000000000001e-05 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.224007+0000", 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20158272999999999 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.224289+0000", 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00028171899999999999 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.122 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.631748+0000", 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40745914599999999 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.631896+0000", 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000147849 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:57.034776+0000", 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40288009400000002 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.123 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.124 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.124 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.124 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:11.126 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18128752-48 from 1638400 to 2260992 tid 1 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18128752-48 from 2260992 to 2916352 tid 2 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18128752-48 from 2916352 to 3604480 tid 3 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18128752-48 from 3604480 to 3817472 tid 4 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:357: copy_from oid 6 from oid 30 current snap is 33 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:358: rmattr oid 47 current snap is 33 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:358: done (6 left) 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 37 current snap is 33 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:359: seq_num 130 ranges {0=638976,638976=671744,1310720=802816,2113536=81920} 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:359: writing smithi18128752-37 from 0 to 638976 tid 1 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:359: writing smithi18128752-37 from 638976 to 1310720 tid 2 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:359: writing smithi18128752-37 from 1310720 to 2113536 tid 3 2024-04-26T01:25:11.128 INFO:tasks.rados.rados.0.smithi181.stdout:359: writing smithi18128752-37 from 2113536 to 2195456 tid 4 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 36 current snap is 33 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:360: seq_num 131 ranges {32768=655360,688128=425984,1114112=638976,1753088=491520} 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:360: writing smithi18128752-36 from 32768 to 688128 tid 1 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:360: writing smithi18128752-36 from 688128 to 1114112 tid 2 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:360: writing smithi18128752-36 from 1114112 to 1753088 tid 3 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:360: writing smithi18128752-36 from 1753088 to 2244608 tid 4 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:361: copy_from oid 38 from oid 45 current snap is 33 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 24 current snap is 33 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:362: seq_num 132 ranges {3244032=671744,3915776=573440,4489216=753664,5242880=229376} 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18128752-24 from 3244032 to 3915776 tid 1 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18128752-24 from 3915776 to 4489216 tid 2 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18128752-24 from 4489216 to 5242880 tid 3 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18128752-24 from 5242880 to 5472256 tid 4 2024-04-26T01:25:11.129 INFO:tasks.rados.rados.0.smithi181.stdout:363: snap_remove snap 31 2024-04-26T01:25:11.130 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:11.127+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:11.227 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:11.231+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:11.267 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:11.267 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:11.267 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:25:11.267 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:11.282 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:25:11.327 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:11.331+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:11.428 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:11.431+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:11.459 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:11.459 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:11.459 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:25:11.459 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:25:11.459 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:11.474 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:25:11.529 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:11.530+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:11.631 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:11.635+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:11.656 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:11.656 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:25:11.657 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:25:11.657 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:11.657 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.657 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.6s4 35/34 ECSubRead(tid=8, to_read={3:60f0095a:::smithi18128752-3:head=0,262144,0}, subchunks={3:60f0095a:::smithi18128752-3:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:11.657 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 65.238494187000001, 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100734536, 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.658 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420050+0000", 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420052+0000", 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.846e-06 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420069+0000", 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7493e-05 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.659 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420098+0000", 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9079000000000001e-05 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.420118+0000", 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9468999999999999e-05 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.660 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520369+0000", 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100251189 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.520785+0000", 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00041545999999999998 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.661 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDPGPush(3.es6 93/90 [PushOp(3:734f3d85:::smithi18128752-49:head, version: 87'73, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(3:734f3d85:::smithi18128752-49:head@87'73, size: 2736128, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:05.419468+0000", 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.2390757649999999, 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101662787, 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.662 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.419468+0000", 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.419468+0000", 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.419470+0000", 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8029999999999999e-06 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.663 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.419885+0000", 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000414965 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.419931+0000", 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5872999999999999e-05 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.419959+0000", 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8113e-05 2024-04-26T01:25:11.664 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.520272+0000", 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10031256700000001 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.521131+0000", 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00085946600000000003 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.665 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDPGPush(3.cs5 93/90 [PushOp(3:3a0c1dcc:::smithi18128752-50:head, version: 87'88, data_included: [524288~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:3a0c1dcc:::smithi18128752-50:head@87'88, size: 3457024, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:05.524559+0000", 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.1339855649999997, 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10148724100000001, 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.524559+0000", 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.666 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.524559+0000", 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.524561+0000", 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.99e-06 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.524932+0000", 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00037105800000000001 2024-04-26T01:25:11.667 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.524978+0000", 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6600000000000001e-05 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.625134+0000", 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10015573 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.625361+0000", 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022732199999999999 2024-04-26T01:25:11.668 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.626046+0000", 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00068454099999999999 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDPGPush(3.es6 93/90 [PushOp(3:734f3d85:::smithi18128752-49:head, version: 87'73, data_included: [524288~159744], data_size: 159744, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:734f3d85:::smithi18128752-49:head@87'73, size: 2736128, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2736128, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:05.631015+0000", 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 6.0275290879999996, 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101706134, 2024-04-26T01:25:11.669 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.631015+0000", 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.670 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.631015+0000", 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.631017+0000", 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.807e-06 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.631246+0000", 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022910500000000001 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.631293+0000", 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6826999999999997e-05 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.631322+0000", 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8907e-05 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.731665+0000", 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100343223 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.672 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.732721+0000", 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001056265 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.1s5 94/90 ECSubWrite(tid=315, reqid=osd.4.0:315, at_version=94'50, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:05.902040+0000", 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.7565038230000001, 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202239486, 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.673 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.902040+0000", 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.902040+0000", 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.674 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.902041+0000", 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4000000000000002e-07 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.902092+0000", 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1127e-05 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:05.902125+0000", 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2886000000000003e-05 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.675 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.676 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.002222+0000", 2024-04-26T01:25:11.676 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10009736900000001 2024-04-26T01:25:11.676 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.676 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.676 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.102434+0000", 2024-04-26T01:25:11.677 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10021195099999999 2024-04-26T01:25:11.677 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.677 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.677 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.677 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.102452+0000", 2024-04-26T01:25:11.678 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8037e-05 2024-04-26T01:25:11.678 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.678 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.678 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.678 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.104118+0000", 2024-04-26T01:25:11.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001665623 2024-04-26T01:25:11.679 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.679 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.679 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.679 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.104280+0000", 2024-04-26T01:25:11.679 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000161953 2024-04-26T01:25:11.680 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.680 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.680 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.680 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.680 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.680 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 94/90 ECSubWrite(tid=139, reqid=osd.11.0:135, at_version=0'0, trim_to=0'0, roll_forward_to=94'90))", 2024-04-26T01:25:11.681 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:06.005692+0000", 2024-04-26T01:25:11.681 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.6528524039999999, 2024-04-26T01:25:11.681 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202257298, 2024-04-26T01:25:11.681 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.681 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.681 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.682 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.682 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.682 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.682 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.005692+0000", 2024-04-26T01:25:11.682 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.682 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.683 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.683 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.005692+0000", 2024-04-26T01:25:11.683 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.683 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.683 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.005693+0000", 2024-04-26T01:25:11.684 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8299999999999995e-07 2024-04-26T01:25:11.684 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.684 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.684 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.005712+0000", 2024-04-26T01:25:11.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9596999999999999e-05 2024-04-26T01:25:11.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.685 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.685 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.685 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.005726+0000", 2024-04-26T01:25:11.685 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3684e-05 2024-04-26T01:25:11.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.686 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.686 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.686 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.105832+0000", 2024-04-26T01:25:11.686 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100106073 2024-04-26T01:25:11.686 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.687 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.206483+0000", 2024-04-26T01:25:11.687 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10065072999999999 2024-04-26T01:25:11.687 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.687 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.687 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.206511+0000", 2024-04-26T01:25:11.688 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8600000000000001e-05 2024-04-26T01:25:11.688 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.688 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.688 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.688 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.207852+0000", 2024-04-26T01:25:11.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013402850000000001 2024-04-26T01:25:11.689 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.689 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.689 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.689 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.207949+0000", 2024-04-26T01:25:11.689 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.7345999999999999e-05 2024-04-26T01:25:11.690 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.690 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.690 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.690 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.690 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.es6 94/90 ECSubWrite(tid=341, reqid=osd.13.0:339, at_version=0'0, trim_to=0'0, roll_forward_to=94'75))", 2024-04-26T01:25:11.691 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:06.106604+0000", 2024-04-26T01:25:11.691 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.55194017, 2024-04-26T01:25:11.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20207259699999999, 2024-04-26T01:25:11.691 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.691 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.691 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.691 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.106604+0000", 2024-04-26T01:25:11.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.106604+0000", 2024-04-26T01:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.106605+0000", 2024-04-26T01:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.7300000000000005e-07 2024-04-26T01:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.106627+0000", 2024-04-26T01:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1739999999999999e-05 2024-04-26T01:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.106658+0000", 2024-04-26T01:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0979999999999998e-05 2024-04-26T01:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.696 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.696 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.106680+0000", 2024-04-26T01:25:11.696 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2793e-05 2024-04-26T01:25:11.696 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.696 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.697 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.306951+0000", 2024-04-26T01:25:11.697 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20027086199999999 2024-04-26T01:25:11.697 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.697 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.697 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.306979+0000", 2024-04-26T01:25:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7603e-05 2024-04-26T01:25:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.308584+0000", 2024-04-26T01:25:11.698 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00160546 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.308677+0000", 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2286000000000004e-05 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.699 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.1s5 94/90 ECSubWrite(tid=317, reqid=osd.4.0:315, at_version=0'0, trim_to=0'0, roll_forward_to=94'50))", 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:06.116579+0000", 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.5419648529999996, 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101412615, 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.700 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116579+0000", 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.701 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116579+0000", 2024-04-26T01:25:11.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.702 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.702 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.702 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.702 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116581+0000", 2024-04-26T01:25:11.702 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5600000000000001e-06 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116601+0000", 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0397000000000001e-05 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116639+0000", 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.82e-05 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.703 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116655+0000", 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5852999999999999e-05 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116856+0000", 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020043200000000001 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.116888+0000", 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2447000000000003e-05 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.704 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.217926+0000", 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10103743599999999 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.217992+0000", 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.6290000000000004e-05 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.705 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDPGRecoveryDelete(3.8s2 e94,91 3:18914e2a:::smithi18128752-43:2,41'18)", 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:06.237689+0000", 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.4208552279999997, 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100502482, 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.237689+0000", 2024-04-26T01:25:11.706 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.237689+0000", 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.237691+0000", 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.854e-06 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.707 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.237702+0000", 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1039999999999999e-05 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.237731+0000", 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9476e-05 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.237747+0000", 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5684000000000001e-05 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.708 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.337964+0000", 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100217509 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.337982+0000", 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7079999999999999e-05 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.709 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.338191+0000", 2024-04-26T01:25:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020983899999999999 2024-04-26T01:25:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.710 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDPGRecoveryDelete(3.8s2 e94,91 3:1945891d:::smithi18128752-34:7,48'34)", 2024-04-26T01:25:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:06.442783+0000", 2024-04-26T01:25:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 5.2157612450000004, 2024-04-26T01:25:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200708363, 2024-04-26T01:25:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.711 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:11.712 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.712 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.442783+0000", 2024-04-26T01:25:11.712 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.712 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.713 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.442783+0000", 2024-04-26T01:25:11.713 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.713 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.713 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.713 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.442785+0000", 2024-04-26T01:25:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:25:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.442796+0000", 2024-04-26T01:25:11.714 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1250000000000001e-05 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.442826+0000", 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0540000000000002e-05 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.715 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.442842+0000", 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5903e-05 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.643243+0000", 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20040054800000001 2024-04-26T01:25:11.716 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.643259+0000", 2024-04-26T01:25:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6290000000000002e-05 2024-04-26T01:25:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.717 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.718 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.718 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.643491+0000", 2024-04-26T01:25:11.718 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023208899999999999 2024-04-26T01:25:11.718 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.718 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.718 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.718 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.719 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.719 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s2 95/91 ECSubWrite(tid=159, reqid=client.4589.0:829, at_version=95'120, trim_to=0'0, roll_forward_to=0'0, backfill_or_async_recovery))", 2024-04-26T01:25:11.719 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:06.894657+0000", 2024-04-26T01:25:11.719 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.7638869259999996, 2024-04-26T01:25:11.719 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.064368508000000005, 2024-04-26T01:25:11.719 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.894657+0000", 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.720 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.894657+0000", 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.894658+0000", 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8000000000000001e-07 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.894698+0000", 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0538999999999999e-05 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.721 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.894722+0000", 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3620000000000001e-05 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.894740+0000", 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7983999999999999e-05 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.958231+0000", 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.063491368000000006 2024-04-26T01:25:11.722 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.958242+0000", 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1243999999999999e-05 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.958989+0000", 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00074677499999999998 2024-04-26T01:25:11.723 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.959026+0000", 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.6597999999999998e-05 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpRead(3.bs3 95/90 ECSubRead(tid=319, to_read={3:d65fea44:::smithi18128752-20:head=0,364544,0}, subchunks={3:d65fea44:::smithi18128752-20:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:06.979709+0000", 2024-04-26T01:25:11.724 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.6788346929999998, 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10077082599999999, 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.979709+0000", 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.979709+0000", 2024-04-26T01:25:11.725 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.979710+0000", 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.3799999999999996e-07 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.979721+0000", 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0210999999999999e-05 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.726 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:06.979740+0000", 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9181000000000002e-05 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.079842+0000", 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100101729 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.080036+0000", 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019432600000000001 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.727 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.080480+0000", 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00044444100000000001 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s2 95/91 ECSubWrite(tid=165, reqid=client.4589.0:833, at_version=0'0, trim_to=0'0, roll_forward_to=95'124, backfill_or_async_recovery))", 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:07.527736+0000", 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.1308079720000004, 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10138040399999999, 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.728 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.527736+0000", 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.527736+0000", 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.527738+0000", 2024-04-26T01:25:11.729 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.21e-06 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.527771+0000", 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2240000000000003e-05 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.527803+0000", 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2679999999999999e-05 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.730 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.527829+0000", 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.5660000000000002e-05 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.628111+0000", 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100281601 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.628130+0000", 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9843999999999998e-05 2024-04-26T01:25:11.731 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.629063+0000", 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00093276899999999996 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.629117+0000", 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3399999999999997e-05 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.732 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s2 95/91 ECSubWrite(tid=166, reqid=client.4589.0:834, at_version=95'125, trim_to=0'0, roll_forward_to=95'124, backfill_or_async_recovery))", 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:07.530507+0000", 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 4.1280376360000002, 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099364981000000005, 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.530507+0000", 2024-04-26T01:25:11.733 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.530507+0000", 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.530508+0000", 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.556e-06 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.734 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.530546+0000", 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7833000000000002e-05 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.530575+0000", 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9102999999999999e-05 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.530599+0000", 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3726999999999998e-05 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.735 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.628437+0000", 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.097837994999999997 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.628451+0000", 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4063000000000001e-05 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.629801+0000", 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001349928 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.736 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:07.629872+0000", 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.0776000000000001e-05 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 98/90 ECSubWrite(tid=148, reqid=client.4589.0:871, at_version=98'94, trim_to=0'0, roll_forward_to=94'90))", 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.785112+0000", 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.87343193900000005, 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20139827799999999, 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.737 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.785112+0000", 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.785112+0000", 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.785114+0000", 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.587e-06 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.785453+0000", 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00033940199999999999 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.785474+0000", 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0475e-05 2024-04-26T01:25:11.739 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885608+0000", 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10013385700000001 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.886032+0000", 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00042455200000000002 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.740 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.886053+0000", 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0613000000000001e-05 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.986425+0000", 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100372195 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.741 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.986511+0000", 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5597000000000001e-05 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDPGPush(3.8s2 98/91 [PushOp(3:14c0c421:::smithi18128752-31:19, version: 87'115, data_included: [786432~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:14c0c421:::smithi18128752-31:19@87'115, size: 4276224, copy_subset: [], clone_subset: {}, snapset: 20=[]:{19: [19], 20: [20, 1f, 1e, 1d]}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.822565+0000", 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.83597885800000005, 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100987913, 2024-04-26T01:25:11.742 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.743 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.743 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.743 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.743 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.743 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.743 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.822565+0000", 2024-04-26T01:25:11.743 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.744 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.744 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.744 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.822565+0000", 2024-04-26T01:25:11.744 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.744 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.822566+0000", 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.3600000000000003e-07 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.822754+0000", 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018809999999999999 2024-04-26T01:25:11.745 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.822775+0000", 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0877e-05 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.822792+0000", 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7402e-05 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.746 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.922992+0000", 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10019958499999999 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.923553+0000", 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00056121299999999997 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.747 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 98/90 ECSubWrite(tid=149, reqid=client.4589.0:872, at_version=98'95, trim_to=0'0, roll_forward_to=94'90))", 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.885666+0000", 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.77287768099999998, 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100954533, 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885666+0000", 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.748 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885666+0000", 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885668+0000", 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3599999999999999e-06 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885865+0000", 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019706899999999999 2024-04-26T01:25:11.749 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885889+0000", 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.457e-05 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885903+0000", 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3390000000000001e-05 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.886992+0000", 2024-04-26T01:25:11.750 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001089455 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.887006+0000", 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.402e-05 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.986552+0000", 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099545945999999996 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.751 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.986621+0000", 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8722999999999999e-05 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.753 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs5 98/90 ECSubWrite(tid=150, reqid=client.4589.0:873, at_version=98'96, trim_to=0'0, roll_forward_to=94'90))", 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.885928+0000", 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.77261614899999997, 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101271815, 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885928+0000", 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.754 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885928+0000", 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885929+0000", 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.44e-07 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885950+0000", 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1472999999999999e-05 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.755 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885991+0000", 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0732999999999998e-05 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.886002+0000", 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.076e-05 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.985861+0000", 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099858980999999999 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.756 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.985899+0000", 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.8068999999999998e-05 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.987125+0000", 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012259580000000001 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.757 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.987200+0000", 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.5197000000000005e-05 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.1s5 99/90 ECSubWrite(tid=327, reqid=client.4589.0:894, at_version=99'52, trim_to=0'0, roll_forward_to=94'50))", 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:11.155998+0000", 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.50254638299999999, 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103558069, 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.758 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.155998+0000", 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.155998+0000", 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.759 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.155999+0000", 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1850000000000001e-06 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.156126+0000", 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000126561 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.760 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.156150+0000", 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.4343999999999999e-05 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.156165+0000", 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5390999999999999e-05 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.256373+0000", 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10020725599999999 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.761 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.256390+0000", 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7125999999999998e-05 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.259495+0000", 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0031057799999999998 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.259556+0000", 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0426e-05 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.762 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s2 99/91 ECSubWrite(tid=185, reqid=client.4589.0:886, at_version=99'133, trim_to=0'0, roll_forward_to=98'130))", 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:11.315226+0000", 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.34331803599999999, 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10189013399999999, 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.315226+0000", 2024-04-26T01:25:11.763 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.315226+0000", 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.315227+0000", 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5199999999999995e-07 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.315532+0000", 2024-04-26T01:25:11.764 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030473099999999998 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.315547+0000", 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5673999999999999e-05 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.315556+0000", 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.4579999999999993e-06 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.415737+0000", 2024-04-26T01:25:11.765 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10018137000000001 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.415749+0000", 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1559999999999999e-05 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.417072+0000", 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001323622 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:11.766 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.417116+0000", 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.3767000000000003e-05 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:11.767 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:11.769 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:25:11.769 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:11.771+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:11.870 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:11.875+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:11.907 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:11.907 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:25:11.907 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 0 2024-04-26T01:25:11.907 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:11.921 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:25:11.971 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:11.972+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:12.008 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:350: finishing write tid 2 to smithi18128752-50 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:350: finishing write tid 3 to smithi18128752-50 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:349: finishing rollback tid 1 to smithi18128752-34 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 0 (ObjNum 49 snap 0 seq_num 3899831938) dirty dne 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing rollback tid 0 to smithi18128752-43 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:360: finishing write tid 1 to smithi18128752-36 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:350: finishing write tid 4 to smithi18128752-50 2024-04-26T01:25:12.010 INFO:tasks.rados.rados.0.smithi181.stdout:350: finishing write tid 5 to smithi18128752-50 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:359: finishing write tid 1 to smithi18128752-37 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:360: finishing write tid 2 to smithi18128752-36 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:354: finishing rollback tid 0 to smithi18128752-40 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:360: finishing write tid 3 to smithi18128752-36 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:357: finishing copy_from racing read to smithi18128752-6 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 1 to smithi18128752-24 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:354: finishing rollback tid 1 to smithi18128752-40 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:354: finishing rollback tid 2 to smithi18128752-40 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 163 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:361: finishing copy_from racing read to smithi18128752-38 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:359: finishing write tid 2 to smithi18128752-37 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:359: finishing write tid 3 to smithi18128752-37 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:359: finishing write tid 4 to smithi18128752-37 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:360: finishing write tid 4 to smithi18128752-36 2024-04-26T01:25:12.011 INFO:tasks.rados.rados.0.smithi181.stdout:360: finishing write tid 5 to smithi18128752-36 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 2 to smithi18128752-24 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:359: finishing write tid 5 to smithi18128752-37 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:357: finishing copy_from to smithi18128752-6 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 74 (ObjNum 115 snap 24 seq_num 115) dirty exists 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 1 to smithi18128752-48 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 2 to smithi18128752-48 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 3 to smithi18128752-48 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 4 to smithi18128752-48 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 5 to smithi18128752-48 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:361: finishing copy_from to smithi18128752-38 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 148 (ObjNum 121 snap 30 seq_num 121) dirty exists 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:359: finishing write tid 6 to smithi18128752-37 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:359: oid 37 updating version 0 to 55 2024-04-26T01:25:12.012 INFO:tasks.rados.rados.0.smithi181.stdout:359: oid 37 version 55 is already newer than 52 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:359: oid 37 version 55 is already newer than 54 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:359: oid 37 version 55 is already newer than 51 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:359: oid 37 version 55 is already newer than 53 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 55 (ObjNum 130 snap 33 seq_num 130) dirty exists 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:359: left oid 37 (ObjNum 130 snap 33 seq_num 130) 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:360: finishing write tid 6 to smithi18128752-36 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:360: oid 36 updating version 0 to 84 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:360: oid 36 version 84 is already newer than 82 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:360: oid 36 version 84 is already newer than 83 2024-04-26T01:25:12.013 INFO:tasks.rados.rados.0.smithi181.stdout:360: oid 36 updating version 84 to 85 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:360: oid 36 version 85 is already newer than 81 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 85 (ObjNum 131 snap 33 seq_num 131) dirty exists 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:360: left oid 36 (ObjNum 131 snap 33 seq_num 131) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:349: done (10 left) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:354: done (9 left) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:357: done (8 left) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:359: done (7 left) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:360: done (6 left) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:361: done (5 left) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:363: done (4 left) 2024-04-26T01:25:12.014 INFO:tasks.rados.rados.0.smithi181.stdout:364: snap_create 2024-04-26T01:25:12.072 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:12.075+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:12.095 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:12.095 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:25:12.095 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:25:12.095 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:25:12.095 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:12.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:25:12.155 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 7, 12] out_osds: [2] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:25:12.155 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:25:12.155 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-26T01:25:12.156 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:25:12.173 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:12.176+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/30 ECSubWrite(tid=11, reqid=client.4589.0:127, at_version=35'7, trim_to=0'0, roll_forward_to=34'6))", 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:25:12.239 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 66.664606801999994, 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202990695, 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.240 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:25:12.241 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.241 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577857+0000", 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.577859+0000", 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.013e-06 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.242 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578257+0000", 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039834700000000001 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578289+0000", 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2016e-05 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.578306+0000", 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6460000000000002e-05 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.243 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678641+0000", 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100335199 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.678656+0000", 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5233e-05 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780779+0000", 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102122955 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.244 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.780847+0000", 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.8472000000000004e-05 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 35/30 ECSubWrite(tid=31, reqid=client.4589.0:124, at_version=35'13, trim_to=0'0, roll_forward_to=34'10))", 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 66.391085548999996, 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20482444299999999, 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.245 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851378+0000", 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.246 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.851380+0000", 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.973e-06 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852052+0000", 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00067198400000000004 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.852084+0000", 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2298999999999997e-05 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952200+0000", 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100116018 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952352+0000", 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015207299999999999 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.952377+0000", 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4363e-05 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.248 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056116+0000", 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103738862 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.056202+0000", 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.6871000000000002e-05 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 35/34 ECSubWrite(tid=18, reqid=client.4589.0:147, at_version=35'10, trim_to=0'0, roll_forward_to=35'4))", 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 65.965378275000006, 2024-04-26T01:25:12.249 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20597036399999999, 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277085+0000", 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.250 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277086+0000", 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8999999999999996e-07 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277250+0000", 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016446899999999999 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277265+0000", 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.418e-05 2024-04-26T01:25:12.251 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.277275+0000", 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0456e-05 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481015+0000", 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203739686 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.481042+0000", 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.6962999999999999e-05 2024-04-26T01:25:12.252 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.482955+0000", 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001913816 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.483056+0000", 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000100204 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.253 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 47/44 ECSubWrite(tid=71, reqid=client.4589.0:315, at_version=0'0, trim_to=0'0, roll_forward_to=46'20))", 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 53.987029335000003, 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30303612499999999, 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.254 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255434+0000", 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255436+0000", 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.81e-06 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255469+0000", 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2895999999999998e-05 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.255 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.255499+0000", 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0253e-05 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.355601+0000", 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100101499 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455871+0000", 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10027049 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.256 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.455899+0000", 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7636000000000001e-05 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558401+0000", 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102502418 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:18.558470+0000", 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9122999999999995e-05 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.257 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=68, reqid=client.4589.0:383, at_version=51'42, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.923915031999996, 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.302538633, 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.258 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318549+0000", 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318551+0000", 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0930000000000001e-06 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318918+0000", 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036699299999999997 2024-04-26T01:25:12.259 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.318977+0000", 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.9203999999999998e-05 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419092+0000", 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100115619 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419200+0000", 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010721300000000001 2024-04-26T01:25:12.260 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419220+0000", 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0363000000000001e-05 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620972+0000", 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.201752242 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621087+0000", 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000114906 2024-04-26T01:25:12.261 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=206, reqid=client.4589.0:388, at_version=51'18, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.900382692999997, 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.300923198, 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.262 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342081+0000", 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342083+0000", 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.793e-06 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.263 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342388+0000", 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030580700000000001 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342408+0000", 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9683000000000001e-05 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.342417+0000", 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2550000000000003e-06 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.442698+0000", 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100280522 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.442716+0000", 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7808999999999998e-05 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.642922+0000", 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20020643499999999 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.265 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.643004+0000", 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1893999999999999e-05 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=69, reqid=client.4589.0:384, at_version=51'43, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.823327618999997, 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20207976799999999, 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.266 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419136+0000", 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419137+0000", 2024-04-26T01:25:12.267 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.54e-07 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419193+0000", 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6375999999999997e-05 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419206+0000", 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.323e-05 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419217+0000", 2024-04-26T01:25:12.268 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0906e-05 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519979+0000", 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10076136199999999 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519989+0000", 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0507e-05 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621131+0000", 2024-04-26T01:25:12.269 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10114197 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621216+0000", 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4662999999999999e-05 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 51/44 ECSubWrite(tid=70, reqid=client.4589.0:385, at_version=51'44, trim_to=0'0, roll_forward_to=48'39))", 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.823227408999998, 2024-04-26T01:25:12.270 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20264654900000001, 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419236+0000", 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.271 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419237+0000", 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8400000000000005e-07 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419295+0000", 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.8211999999999998e-05 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.419307+0000", 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2558000000000001e-05 2024-04-26T01:25:12.272 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.519470+0000", 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100162563 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620565+0000", 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10109543 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.620584+0000", 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8352e-05 2024-04-26T01:25:12.273 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621809+0000", 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012251160000000001 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.621883+0000", 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3833999999999998e-05 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.274 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=207, reqid=client.4589.0:389, at_version=51'19, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.801044603000001, 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202618775, 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.275 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441419+0000", 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441420+0000", 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1489999999999999e-06 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441473+0000", 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2834999999999998e-05 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.276 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441492+0000", 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9145000000000001e-05 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.441503+0000", 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1151e-05 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.641827+0000", 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200323321 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.277 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.641854+0000", 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7695000000000002e-05 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.643949+0000", 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020951979999999999 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.644038+0000", 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.8281000000000004e-05 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.278 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.2s5 51/44 ECSubWrite(tid=209, reqid=client.4589.0:390, at_version=0'0, trim_to=0'0, roll_forward_to=51'20))", 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 48.394201266000003, 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.378972751, 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.279 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848262+0000", 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848264+0000", 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3510000000000001e-06 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848286+0000", 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2011999999999999e-05 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.280 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848307+0000", 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1798999999999999e-05 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848322+0000", 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4968e-05 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848406+0000", 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3762000000000002e-05 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.281 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.848418+0000", 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1626e-05 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.227182+0000", 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.37876395099999999 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.227235+0000", 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.3282000000000002e-05 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.282 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.283 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/44 ECSubWrite(tid=146, reqid=osd.14.0:146, at_version=67'60, trim_to=0'0, roll_forward_to=61'58))", 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.452141249999997, 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20292310799999999, 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.284 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790322+0000", 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790323+0000", 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.1299999999999999e-07 2024-04-26T01:25:12.285 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790372+0000", 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8862000000000001e-05 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790402+0000", 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9595999999999999e-05 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.790435+0000", 2024-04-26T01:25:12.286 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3872999999999999e-05 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.891521+0000", 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10108587600000001 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.891538+0000", 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6816999999999998e-05 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.993158+0000", 2024-04-26T01:25:12.287 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101619426 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.993245+0000", 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7844999999999997e-05 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpRead(3.8s2 67/44 ECSubRead(tid=111, to_read={3:1fe6984d:::smithi18128752-48:head=0,811008,0}, subchunks={3:1fe6984d:::smithi18128752-48:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 33.445167562999998, 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.296535041, 2024-04-26T01:25:12.288 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797296+0000", 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.289 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797298+0000", 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.66e-06 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797315+0000", 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7555999999999999e-05 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797346+0000", 2024-04-26T01:25:12.290 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.1192999999999998e-05 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:38.797375+0000", 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8484999999999999e-05 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.093325+0000", 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.29594971799999997 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.291 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.093831+0000", 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00050642899999999997 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 67/44 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=67'64, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 32.907445721000002, 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20216706400000001, 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.292 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335018+0000", 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3700000000000001e-07 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.293 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335038+0000", 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0047e-05 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.335047+0000", 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.7329999999999992e-06 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.435145+0000", 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100097939 2024-04-26T01:25:12.294 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.535402+0000", 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10025746200000001 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.535419+0000", 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6529e-05 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.537144+0000", 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0017251930000000001 2024-04-26T01:25:12.295 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:39.537185+0000", 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0824000000000001e-05 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.6s2 72/63 ECSubWrite(tid=54, reqid=osd.0.0:54, at_version=72'37, trim_to=0'0, roll_forward_to=71'35))", 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 28.330680750999999, 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20240396699999999, 2024-04-26T01:25:12.296 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.297 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911783+0000", 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7000000000000004e-07 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911840+0000", 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.6796000000000003e-05 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911867+0000", 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.7155999999999999e-05 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.298 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911891+0000", 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3195999999999999e-05 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.112540+0000", 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.200648943 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.112562+0000", 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2189999999999999e-05 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.299 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.114107+0000", 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0015455899999999999 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.114187+0000", 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9425999999999995e-05 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.8s2 72/44 ECSubWrite(tid=131, reqid=osd.5.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=72'94))", 2024-04-26T01:25:12.300 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:44.779998+0000", 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 27.462465919, 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.20524366399999999, 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.779998+0000", 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.779998+0000", 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.301 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.780000+0000", 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.883e-06 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.780028+0000", 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8333000000000001e-05 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.780066+0000", 2024-04-26T01:25:12.302 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.7942999999999999e-05 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.880189+0000", 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100123209 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.982504+0000", 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102314949 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.982519+0000", 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5167000000000001e-05 2024-04-26T01:25:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.985126+0000", 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0026065659999999998 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.985241+0000", 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011561399999999999 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.304 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.es2 78/44 ECSubWrite(tid=295, reqid=client.4589.0:685, at_version=78'63, trim_to=0'0, roll_forward_to=72'61))", 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:50.188947+0000", 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 22.053516876, 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202856176, 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.188947+0000", 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.188947+0000", 2024-04-26T01:25:12.305 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.188948+0000", 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7540000000000001e-06 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.189225+0000", 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000276758 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.306 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.189258+0000", 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3195999999999999e-05 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.189292+0000", 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3692999999999997e-05 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.289603+0000", 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100310776 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.289619+0000", 2024-04-26T01:25:12.307 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6013000000000001e-05 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.391740+0000", 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10212062399999999 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:50.391803+0000", 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3361999999999994e-05 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.308 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.4s3 88/44 ECSubWrite(tid=90, reqid=client.4589.0:789, at_version=88'58, trim_to=0'0, roll_forward_to=87'55))", 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:59.915834+0000", 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 12.326629348000001, 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.204355125, 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:59.915834+0000", 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.309 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:59.915834+0000", 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:59.915836+0000", 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9860000000000001e-06 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:59.916129+0000", 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029287900000000001 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.310 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:59.916162+0000", 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3189999999999999e-05 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:00.016293+0000", 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100130227 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:00.116975+0000", 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100682457 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.311 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:00.116991+0000", 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6339999999999999e-05 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:00.120083+0000", 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0030918270000000001 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:00.120189+0000", 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010621899999999999 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.312 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 94/44 ECSubWrite(tid=260, reqid=osd.14.0:260, at_version=94'140, trim_to=0'0, roll_forward_to=87'138))", 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:05.803100+0000", 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.4393631459999998, 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.205576906, 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.803100+0000", 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.313 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.803100+0000", 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.803101+0000", 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.5599999999999995e-07 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.803145+0000", 2024-04-26T01:25:12.314 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.4156000000000002e-05 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.803174+0000", 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.8649999999999998e-05 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.903281+0000", 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10010716 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.903527+0000", 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000245617 2024-04-26T01:25:12.315 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.903544+0000", 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7856999999999999e-05 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:06.008580+0000", 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105035984 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:06.008677+0000", 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6926e-05 2024-04-26T01:25:12.316 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s3 94/44 ECSubWrite(tid=265, reqid=client.4589.0:822, at_version=94'147, trim_to=0'0, roll_forward_to=87'138))", 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:05.935436+0000", 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 6.3070274700000004, 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.27241367700000002, 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.317 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.935436+0000", 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.935436+0000", 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.935438+0000", 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.318 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.935484+0000", 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.6505999999999998e-05 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.935515+0000", 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0297000000000001e-05 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:05.935531+0000", 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6460000000000002e-05 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.319 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:06.205370+0000", 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.26983870500000001 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:06.205385+0000", 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5630000000000001e-05 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:06.207742+0000", 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0023570039999999998 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.320 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:06.207850+0000", 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00010733199999999999 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "pg_update_log_missing(3.8s3 epoch 98/91 rep_tid 180 entries 98'128 (0'0) error 3:1945891d:::smithi18128752-34:head by client.4589.0:868 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 67'76)", 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:10.620170+0000", 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.622293209, 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.202500455, 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "reached pg", 2024-04-26T01:25:12.321 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.620170+0000", 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.620170+0000", 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.620173+0000", 2024-04-26T01:25:12.322 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.424e-06 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.620196+0000", 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3105e-05 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.620259+0000", 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.3442999999999995e-05 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.720396+0000", 2024-04-26T01:25:12.323 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100136906 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.820668+0000", 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10027209199999999 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.822671+0000", 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.002002485 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.324 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.325 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:350: finishing write tid 6 to smithi18128752-50 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:350: oid 50 updating version 0 to 94 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:350: oid 50 version 94 is already newer than 92 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:350: oid 50 updating version 94 to 95 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:350: oid 50 updating version 95 to 96 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:350: oid 50 version 96 is already newer than 93 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 96 (ObjNum 128 snap 32 seq_num 128) dirty exists 2024-04-26T01:25:12.326 INFO:tasks.rados.rados.0.smithi181.stdout:350: left oid 50 (ObjNum 128 snap 32 seq_num 128) 2024-04-26T01:25:12.327 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:25:12.328 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:12.331+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 6 to smithi18128752-48 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 48 updating version 0 to 135 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 48 updating version 135 to 136 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 48 version 136 is already newer than 133 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 48 version 136 is already newer than 132 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 48 version 136 is already newer than 134 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 136 (ObjNum 129 snap 33 seq_num 129) dirty exists 2024-04-26T01:25:12.344 INFO:tasks.rados.rados.0.smithi181.stdout:356: left oid 48 (ObjNum 129 snap 33 seq_num 129) 2024-04-26T01:25:12.427 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:12.426+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:12.462 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:12.463 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:25:12.480 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-04-26T01:25:12.009451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94749216,"num_objects":51,"num_object_clones":7,"num_object_copies":347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":564,"num_read_kb":141722,"num_write":822,"num_write_kb":139380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":51270176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1147,"ondisk_log_size":1147,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":134,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":637492,"kb_used_data":169720,"kb_used_omap":23,"kb_used_meta":467496,"kb_avail":1414940108,"statfs":{"total":1449551462400,"available":1448898670592,"internally_reserved":0,"allocated":173793280,"data_stored":193950644,"data_compressed":3345600,"data_compressed_allocated":4591616,"data_compressed_original":10055680,"omap_allocated":23926,"internal_metadata":478716554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4997120,"num_objects":-8,"num_object_clones":-7,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-36,"num_objects_misplaced":-5,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":-1,"num_read":14,"num_read_kb":10287,"num_write":11,"num_write_kb":6928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":21364736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.169911"},"pg_stats":[{"pgid":"3.f","version":"96'10","reported_seq":344,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:08.326788+0000","last_change":"2024-04-26T01:25:08.327617+0000","last_active":"2024-04-26T01:25:08.326788+0000","last_peered":"2024-04-26T01:25:08.326788+0000","last_clean":"2024-04-26T01:25:08.326788+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:25:08.326788+0000","last_undegraded":"2024-04-26T01:25:08.326788+0000","last_fullsized":"2024-04-26T01:25:08.326788+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:21.799045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31007455,"stat_sum":{"num_bytes":753664,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":3333,"num_write":6,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"3.e","version":"94'75","reported_seq":370,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.919082+0000","last_change":"2024-04-26T01:25:08.016246+0000","last_active":"2024-04-26T01:25:07.919082+0000","last_peered":"2024-04-26T01:25:07.919082+0000","last_clean":"2024-04-26T01:25:07.919082+0000","last_became_active":"2024-04-26T01:25:03.422052+0000","last_became_peered":"2024-04-26T01:25:03.422052+0000","last_unstale":"2024-04-26T01:25:07.919082+0000","last_undegraded":"2024-04-26T01:25:07.919082+0000","last_fullsized":"2024-04-26T01:25:07.919082+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.169617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098670969999999997,"stat_sum":{"num_bytes":6569984,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":12431,"num_write":34,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,12],"acting":[13,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"3.d","version":"94'74","reported_seq":247,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.907512+0000","last_change":"2024-04-26T01:25:08.008150+0000","last_active":"2024-04-26T01:25:07.907512+0000","last_peered":"2024-04-26T01:25:07.907512+0000","last_clean":"2024-04-26T01:25:07.907512+0000","last_became_active":"2024-04-26T01:24:36.105677+0000","last_became_peered":"2024-04-26T01:24:36.105677+0000","last_unstale":"2024-04-26T01:25:07.907512+0000","last_undegraded":"2024-04-26T01:25:07.907512+0000","last_fullsized":"2024-04-26T01:25:07.907512+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:23.922516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10101033,"stat_sum":{"num_bytes":3342336,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":14699,"num_write":28,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"3.c","version":"94'90","reported_seq":456,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:08.117607+0000","last_change":"2024-04-26T01:25:08.117931+0000","last_active":"2024-04-26T01:25:08.117607+0000","last_peered":"2024-04-26T01:25:08.117607+0000","last_clean":"2024-04-26T01:25:08.117607+0000","last_became_active":"2024-04-26T01:25:03.109927+0000","last_became_peered":"2024-04-26T01:25:03.109927+0000","last_unstale":"2024-04-26T01:25:08.117607+0000","last_undegraded":"2024-04-26T01:25:08.117607+0000","last_fullsized":"2024-04-26T01:25:08.117607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:10.231101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047316699999999998,"stat_sum":{"num_bytes":7192576,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":3851,"num_write":72,"num_write_kb":11312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8650752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"3.b","version":"72'28","reported_seq":403,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.924283+0000","last_change":"2024-04-26T01:25:07.925198+0000","last_active":"2024-04-26T01:25:07.924283+0000","last_peered":"2024-04-26T01:25:07.924283+0000","last_clean":"2024-04-26T01:25:07.924283+0000","last_became_active":"2024-04-26T01:25:03.018610+0000","last_became_peered":"2024-04-26T01:25:03.018610+0000","last_unstale":"2024-04-26T01:25:07.924283+0000","last_undegraded":"2024-04-26T01:25:07.924283+0000","last_fullsized":"2024-04-26T01:25:07.924283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.357679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012495710000000001,"stat_sum":{"num_bytes":1458176,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":4936,"num_write":30,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3407872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"3.a","version":"82'39","reported_seq":373,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:25:06.923224+0000","last_change":"2024-04-26T01:25:06.819466+0000","last_active":"2024-04-26T01:25:06.923224+0000","last_peered":"2024-04-26T01:25:06.923224+0000","last_clean":"2024-04-26T01:25:06.923224+0000","last_became_active":"2024-04-26T01:24:36.102041+0000","last_became_peered":"2024-04-26T01:24:36.102041+0000","last_unstale":"2024-04-26T01:25:06.923224+0000","last_undegraded":"2024-04-26T01:25:06.923224+0000","last_fullsized":"2024-04-26T01:25:06.923224+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'39","last_scrub_stamp":"2024-04-26T01:25:06.819166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:06.819166+0000","objects_scrubbed":1,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:11.965684+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0.001094243,"stat_sum":{"num_bytes":3293184,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":9684,"num_write":33,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"18","length":"3"},{"start":"1c","length":"1"}]},{"pgid":"3.9","version":"91'65","reported_seq":383,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:08.042799+0000","last_change":"2024-04-26T01:25:08.043799+0000","last_active":"2024-04-26T01:25:08.042799+0000","last_peered":"2024-04-26T01:25:08.042799+0000","last_clean":"2024-04-26T01:25:08.042799+0000","last_became_active":"2024-04-26T01:24:01.291346+0000","last_became_peered":"2024-04-26T01:24:01.291346+0000","last_unstale":"2024-04-26T01:25:08.042799+0000","last_undegraded":"2024-04-26T01:25:08.042799+0000","last_fullsized":"2024-04-26T01:25:08.042799+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015667669999999999,"stat_sum":{"num_bytes":5718016,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":1015,"num_write":24,"num_write_kb":7664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"3.8","version":"95'124","reported_seq":490,"reported_epoch":95,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-04-26T01:25:07.523814+0000","last_change":"2024-04-26T01:25:06.888542+0000","last_active":"2024-04-26T01:25:07.523814+0000","last_peered":"2024-04-26T01:25:07.523814+0000","last_clean":"2024-04-26T01:25:01.261092+0000","last_became_active":"2024-04-26T01:25:04.195615+0000","last_became_peered":"2024-04-26T01:25:04.195615+0000","last_unstale":"2024-04-26T01:25:07.523814+0000","last_undegraded":"2024-04-26T01:25:06.887432+0000","last_fullsized":"2024-04-26T01:25:03.788895+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31276866399999997,"stat_sum":{"num_bytes":15335424,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":48,"num_read_kb":26580,"num_write":97,"num_write_kb":20448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":4161536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,12,1,14,15,6],"acting":[5,0,2147483647,2147483647,14,15,6],"avail_no_missing":["5(0)","0(1)","1(2)","6(6)","9(3)","14(4)","15(5)"],"object_location_counts":[{"shards":"0(1),1(2),5(0),6(6),9(3),14(4),15(5)","objects":5},{"shards":"0(1),5(0),6(6),14(4),15(5)","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"18","length":"1"},{"start":"1a","length":"1"}]},{"pgid":"1.0","version":"34'157","reported_seq":369,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.916924+0000","last_change":"2024-04-26T01:24:56.353548+0000","last_active":"2024-04-26T01:25:07.916924+0000","last_peered":"2024-04-26T01:25:07.916924+0000","last_clean":"2024-04-26T01:25:07.916924+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:25:07.916924+0000","last_undegraded":"2024-04-26T01:25:07.916924+0000","last_fullsized":"2024-04-26T01:25:07.916924+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:10.087788+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":147,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:25:06.864563+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:25:06.864563+0000","last_peered":"2024-04-26T01:25:06.864563+0000","last_clean":"2024-04-26T01:25:06.864563+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:06.864563+0000","last_undegraded":"2024-04-26T01:25:06.864563+0000","last_fullsized":"2024-04-26T01:25:06.864563+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.478758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"96'68","reported_seq":409,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:08.226386+0000","last_change":"2024-04-26T01:25:08.125719+0000","last_active":"2024-04-26T01:25:08.226386+0000","last_peered":"2024-04-26T01:25:08.226386+0000","last_clean":"2024-04-26T01:25:08.226386+0000","last_became_active":"2024-04-26T01:24:16.348216+0000","last_became_peered":"2024-04-26T01:24:16.348216+0000","last_unstale":"2024-04-26T01:25:08.226386+0000","last_undegraded":"2024-04-26T01:25:08.226386+0000","last_fullsized":"2024-04-26T01:25:08.226386+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.442874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20842993600000001,"stat_sum":{"num_bytes":12419072,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":1210,"num_write":31,"num_write_kb":12848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"2.0","version":"0'0","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:25:06.922957+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:06.922957+0000","last_peered":"2024-04-26T01:25:06.922957+0000","last_clean":"2024-04-26T01:25:06.922957+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:06.922957+0000","last_undegraded":"2024-04-26T01:25:06.922957+0000","last_fullsized":"2024-04-26T01:25:06.922957+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"94'50","reported_seq":421,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.924122+0000","last_change":"2024-04-26T01:25:08.024941+0000","last_active":"2024-04-26T01:25:07.924122+0000","last_peered":"2024-04-26T01:25:07.924122+0000","last_clean":"2024-04-26T01:25:07.924122+0000","last_became_active":"2024-04-26T01:25:03.216745+0000","last_became_peered":"2024-04-26T01:25:03.216745+0000","last_unstale":"2024-04-26T01:25:07.924122+0000","last_undegraded":"2024-04-26T01:25:07.924122+0000","last_fullsized":"2024-04-26T01:25:07.924122+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:20.193830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10132492,"stat_sum":{"num_bytes":2850816,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":11803,"num_write":27,"num_write_kb":9344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":5357568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,12,13],"acting":[4,11,9,5,10,12,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"2.1","version":"0'0","reported_seq":149,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.917115+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:25:07.917115+0000","last_peered":"2024-04-26T01:25:07.917115+0000","last_clean":"2024-04-26T01:25:07.917115+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:07.917115+0000","last_undegraded":"2024-04-26T01:25:07.917115+0000","last_fullsized":"2024-04-26T01:25:07.917115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:09.755555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"94'148","reported_seq":500,"reported_epoch":96,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T01:25:08.145558+0000","last_change":"2024-04-26T01:25:08.145628+0000","last_active":"2024-04-26T01:25:08.145558+0000","last_peered":"2024-04-26T01:25:08.145558+0000","last_clean":"2024-04-26T01:25:08.145558+0000","last_became_active":"2024-04-26T01:24:16.549243+0000","last_became_peered":"2024-04-26T01:24:16.549243+0000","last_unstale":"2024-04-26T01:25:08.145558+0000","last_undegraded":"2024-04-26T01:25:08.145558+0000","last_fullsized":"2024-04-26T01:25:08.145558+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70893888800000004,"stat_sum":{"num_bytes":7880704,"num_objects":8,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":23878,"num_write":99,"num_write_kb":14208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"18","length":"3"},{"start":"1c","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":160,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.023306+0000","last_change":"2024-04-26T01:25:02.920443+0000","last_active":"2024-04-26T01:25:07.023306+0000","last_peered":"2024-04-26T01:25:07.023306+0000","last_clean":"2024-04-26T01:25:07.023306+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:07.023306+0000","last_undegraded":"2024-04-26T01:25:07.023306+0000","last_fullsized":"2024-04-26T01:25:07.023306+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.351681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"71'11","reported_seq":362,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.918078+0000","last_change":"2024-04-26T01:25:07.918898+0000","last_active":"2024-04-26T01:25:07.918078+0000","last_peered":"2024-04-26T01:25:07.918078+0000","last_clean":"2024-04-26T01:25:07.918078+0000","last_became_active":"2024-04-26T01:24:36.104415+0000","last_became_peered":"2024-04-26T01:24:36.104415+0000","last_unstale":"2024-04-26T01:25:07.918078+0000","last_undegraded":"2024-04-26T01:25:07.918078+0000","last_fullsized":"2024-04-26T01:25:07.918078+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.717632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001443838,"stat_sum":{"num_bytes":901120,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":4532,"num_write":23,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,6,5,3],"acting":[13,7,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"3.5","version":"0'0","reported_seq":338,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.841757+0000","last_change":"2024-04-26T01:25:07.842192+0000","last_active":"2024-04-26T01:25:07.841757+0000","last_peered":"2024-04-26T01:25:07.841757+0000","last_clean":"2024-04-26T01:25:07.841757+0000","last_became_active":"2024-04-26T01:24:36.199837+0000","last_became_peered":"2024-04-26T01:24:36.199837+0000","last_unstale":"2024-04-26T01:25:07.841757+0000","last_undegraded":"2024-04-26T01:25:07.841757+0000","last_fullsized":"2024-04-26T01:25:07.841757+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.427499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064219899999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"2.4","version":"0'0","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:08.023578+0000","last_change":"2024-04-26T01:23:52.270744+0000","last_active":"2024-04-26T01:25:08.023578+0000","last_peered":"2024-04-26T01:25:08.023578+0000","last_clean":"2024-04-26T01:25:08.023578+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:08.023578+0000","last_undegraded":"2024-04-26T01:25:08.023578+0000","last_fullsized":"2024-04-26T01:25:08.023578+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:08.652682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"96'68","reported_seq":422,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-26T01:25:09.040143+0000","last_change":"2024-04-26T01:25:08.133166+0000","last_active":"2024-04-26T01:25:09.040143+0000","last_peered":"2024-04-26T01:25:09.040143+0000","last_clean":"2024-04-26T01:25:09.040143+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:25:09.040143+0000","last_undegraded":"2024-04-26T01:25:09.040143+0000","last_fullsized":"2024-04-26T01:25:09.040143+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:11.708590+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.143991757,"stat_sum":{"num_bytes":14499840,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":15375,"num_write":35,"num_write_kb":14640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":148,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.841913+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:25:07.841913+0000","last_peered":"2024-04-26T01:25:07.841913+0000","last_clean":"2024-04-26T01:25:07.841913+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:07.841913+0000","last_undegraded":"2024-04-26T01:25:07.841913+0000","last_fullsized":"2024-04-26T01:25:07.841913+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"94'103","reported_seq":435,"reported_epoch":96,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T01:25:08.327301+0000","last_change":"2024-04-26T01:25:07.794455+0000","last_active":"2024-04-26T01:25:08.327301+0000","last_peered":"2024-04-26T01:25:08.327301+0000","last_clean":"2024-04-26T01:25:08.327301+0000","last_became_active":"2024-04-26T01:24:01.091412+0000","last_became_peered":"2024-04-26T01:24:01.091412+0000","last_unstale":"2024-04-26T01:25:08.327301+0000","last_undegraded":"2024-04-26T01:25:08.327301+0000","last_fullsized":"2024-04-26T01:25:08.327301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20666024499999999,"stat_sum":{"num_bytes":11829248,"num_objects":4,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":45,"num_read_kb":5632,"num_write":62,"num_write_kb":13216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"18","length":"3"},{"start":"1c","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":150,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-26T01:25:09.039328+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:25:09.039328+0000","last_peered":"2024-04-26T01:25:09.039328+0000","last_clean":"2024-04-26T01:25:09.039328+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:09.039328+0000","last_undegraded":"2024-04-26T01:25:09.039328+0000","last_fullsized":"2024-04-26T01:25:09.039328+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:10.471688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"72'37","reported_seq":413,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.841190+0000","last_change":"2024-04-26T01:25:07.841542+0000","last_active":"2024-04-26T01:25:07.841190+0000","last_peered":"2024-04-26T01:25:07.841190+0000","last_clean":"2024-04-26T01:25:07.841190+0000","last_became_active":"2024-04-26T01:25:03.106948+0000","last_became_peered":"2024-04-26T01:25:03.106948+0000","last_unstale":"2024-04-26T01:25:07.841190+0000","last_undegraded":"2024-04-26T01:25:07.841190+0000","last_fullsized":"2024-04-26T01:25:07.841190+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066944699999999999,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2645,"num_write":18,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"18","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":149,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-26T01:25:07.841577+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:25:07.841577+0000","last_peered":"2024-04-26T01:25:07.841577+0000","last_clean":"2024-04-26T01:25:07.841577+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:07.841577+0000","last_undegraded":"2024-04-26T01:25:07.841577+0000","last_fullsized":"2024-04-26T01:25:07.841577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":94158848,"num_objects":49,"num_object_clones":7,"num_object_copies":343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":428,"num_read_kb":141604,"num_write":619,"num_write_kb":135920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":50741248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163921920,"data_stored":192491520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":990,"ondisk_log_size":990,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280532,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":5536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":5668864,"data_stored":5383268,"data_compressed":223438,"data_compressed_allocated":307200,"data_compressed_original":614400,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280530,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44448,"kb_used_data":13268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327392,"statfs":{"total":96636764160,"available":96591249408,"internally_reserved":0,"allocated":13586432,"data_stored":13446026,"data_compressed":220407,"data_compressed_allocated":303104,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280531,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46964,"kb_used_data":15780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324876,"statfs":{"total":96636764160,"available":96588673024,"internally_reserved":0,"allocated":16158720,"data_stored":18092768,"data_compressed":217386,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313234,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":16180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":16568320,"data_stored":19118986,"data_compressed":220407,"data_compressed_allocated":303104,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313235,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44664,"kb_used_data":13480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327176,"statfs":{"total":96636764160,"available":96591028224,"internally_reserved":0,"allocated":13803520,"data_stored":15745760,"data_compressed":217386,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313235,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313234,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41604,"kb_used_data":10416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330236,"statfs":{"total":96636764160,"available":96594161664,"internally_reserved":0,"allocated":10665984,"data_stored":13785994,"data_compressed":220407,"data_compressed_allocated":303104,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313235,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37564,"kb_used_data":6376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334276,"statfs":{"total":96636764160,"available":96598298624,"internally_reserved":0,"allocated":6529024,"data_stored":8185088,"data_compressed":251441,"data_compressed_allocated":339968,"data_compressed_original":1122304,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313234,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42696,"kb_used_data":11512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329144,"statfs":{"total":96636764160,"available":96593043456,"internally_reserved":0,"allocated":11788288,"data_stored":11649760,"data_compressed":217386,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313235,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47256,"kb_used_data":11912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324584,"statfs":{"total":96636764160,"available":96588374016,"internally_reserved":0,"allocated":12197888,"data_stored":13339230,"data_compressed":214367,"data_compressed_allocated":294912,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313235,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47572,"kb_used_data":16388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324268,"statfs":{"total":96636764160,"available":96588050432,"internally_reserved":0,"allocated":16781312,"data_stored":19903200,"data_compressed":217386,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313235,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30320,"kb_used_data":3296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341520,"statfs":{"total":96636764160,"available":96605716480,"internally_reserved":0,"allocated":3375104,"data_stored":3093216,"data_compressed":217386,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313235,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45412,"kb_used_data":14228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326428,"statfs":{"total":96636764160,"available":96590262272,"internally_reserved":0,"allocated":14569472,"data_stored":15512288,"data_compressed":217386,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313234,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34964,"kb_used_data":7940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336876,"statfs":{"total":96636764160,"available":96600961024,"internally_reserved":0,"allocated":8130560,"data_stored":9450208,"data_compressed":217386,"data_compressed_allocated":299008,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313236,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45160,"kb_used_data":13972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326680,"statfs":{"total":96636764160,"available":96590520320,"internally_reserved":0,"allocated":14307328,"data_stored":15661962,"data_compressed":220407,"data_compressed_allocated":303104,"data_compressed_original":606208,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313235,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44780,"kb_used_data":9436,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94327060,"statfs":{"total":96636764160,"available":96590909440,"internally_reserved":0,"allocated":9662464,"data_stored":11582890,"data_compressed":253024,"data_compressed_allocated":344064,"data_compressed_original":1118208,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15060992,"data_stored":18841600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16494592,"data_stored":19222528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5382144,"data_stored":6840320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":10690560,"data_stored":10698752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11497472,"data_stored":13443072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15683584,"data_stored":19087360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2277376,"data_stored":2277376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12476416,"data_stored":12619776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13471744,"data_stored":14561280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7032832,"data_stored":8634368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":13197312,"data_stored":14835712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4546560,"data_stored":4546560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":9527296,"data_stored":11128832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12705792,"data_stored":16359424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":9555968,"data_stored":14524416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:25:12.483 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-26T01:25:12.483 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-04-26T01:25:12.503 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:12.503 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:25:12.503 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 100/91 ECSubReadReply(tid=195, attrs_read=0))", 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.248986+0000", 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.25903304900000002, 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25905263299999998, 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.248986+0000", 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.504 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.248986+0000", 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.248987+0000", 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.8100000000000002e-07 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.249211+0000", 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000224069 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.505 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.249224+0000", 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3288999999999999e-05 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349343+0000", 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100118779 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 100/91 ECSubReadReply(tid=194, attrs_read=0))", 2024-04-26T01:25:12.506 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.250021+0000", 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.25799844100000002, 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.25804945800000001, 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.250021+0000", 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.250021+0000", 2024-04-26T01:25:12.507 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.250021+0000", 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.250021+0000", 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.348330+0000", 2024-04-26T01:25:12.508 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.098309006000000004 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.348339+0000", 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1730000000000001e-06 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s4 100/44 ECSubWrite(tid=287, reqid=osd.14.0:286, at_version=0'0, trim_to=0'0, roll_forward_to=100'167))", 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.349610+0000", 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.15840870500000001, 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15848821399999999, 2024-04-26T01:25:12.509 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349610+0000", 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349610+0000", 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.510 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349612+0000", 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.55e-06 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349637+0000", 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.4980000000000001e-05 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349670+0000", 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2913000000000001e-05 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.511 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349693+0000", 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3319000000000002e-05 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:879 3.8s0 3:18914e2a:::smithi18128752-43:head [rollback 1d] snapc 21=[21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:10.784784+0000", 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.723235528, 2024-04-26T01:25:12.512 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7233492580000001, 2024-04-26T01:25:12.513 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.513 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.513 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for degraded object", 2024-04-26T01:25:12.513 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.513 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.513 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.513 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 879 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784784+0000", 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784784+0000", 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.514 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784785+0000", 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3540000000000001e-06 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784807+0000", 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1696e-05 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784828+0000", 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.1183e-05 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.515 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784846+0000", 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.8102999999999999e-05 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.789092+0000", 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0042461590000000002 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.789396+0000", 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00030408600000000002 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:12.516 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.789678+0000", 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028145999999999998 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 100/91 ECSubReadReply(tid=195, attrs_read=0))", 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.349751+0000", 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.158267927, 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15842340999999999, 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.517 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349751+0000", 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349751+0000", 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.349752+0000", 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.037e-06 2024-04-26T01:25:12.518 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.350143+0000", 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00039094500000000002 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.350174+0000", 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0858999999999997e-05 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.350192+0000", 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7677e-05 2024-04-26T01:25:12.519 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:880 3.8s0 3:18914e2a:::smithi18128752-43:head [append 0~0] snapc 21=[21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:10.784924+0000", 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.723095123, 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7232810189999999, 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "waiting for degraded object", 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.520 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 880 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784924+0000", 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784924+0000", 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.521 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784925+0000", 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2100000000000001e-06 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784933+0000", 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.3000000000000002e-06 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784951+0000", 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7589999999999999e-05 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.522 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.784960+0000", 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.2969999999999992e-06 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.891346+0000", 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.106385565 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:10.891532+0000", 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000186131 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.523 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s4 100/44 ECSubWrite(tid=288, reqid=osd.14.0:288, at_version=100'169, trim_to=0'0, roll_forward_to=100'167))", 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.351256+0000", 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.15676358300000001, 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15698674900000001, 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.524 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.351256+0000", 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.351256+0000", 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.351257+0000", 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.64e-06 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.525 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.351298+0000", 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1048999999999999e-05 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.351327+0000", 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9116999999999999e-05 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.351343+0000", 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5313000000000001e-05 2024-04-26T01:25:12.526 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.0s4 100/44 ECSubWrite(tid=289, reqid=osd.14.0:289, at_version=100'171, trim_to=0'0, roll_forward_to=100'167))", 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.353135+0000", 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.15488383, 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.155133666, 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.527 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.353135+0000", 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.353135+0000", 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.353137+0000", 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.66e-06 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.528 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.353179+0000", 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1848999999999999e-05 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.353208+0000", 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.9575999999999999e-05 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.353224+0000", 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.573e-05 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.529 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpWrite(3.3s5 100/63 ECSubWrite(tid=358, reqid=osd.13.0:356, at_version=0'0, trim_to=0'0, roll_forward_to=100'15))", 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.423105+0000", 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.084914165999999999, 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.085190323999999998, 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "no flag points reached", 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.423105+0000", 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.530 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.423105+0000", 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.423107+0000", 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.3800000000000001e-06 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.423133+0000", 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5639e-05 2024-04-26T01:25:12.531 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.423164+0000", 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0787000000000002e-05 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDECSubOpReadReply(3.8s0 100/91 ECSubReadReply(tid=195, attrs_read=0))", 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.449888+0000", 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.058131347, 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.058431344000000003, 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.532 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.449888+0000", 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.449888+0000", 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.533 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.449888+0000", 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.449888+0000", 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.449921+0000", 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3138e-05 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.534 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.449933+0000", 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1990999999999999e-05 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "MOSDPGPushReply(3.8s0 100/91 [PushReplyOp(3:14c0c421:::smithi18128752-31:20)])", 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:12.142058+0000", 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 0.36596151399999999, 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.36628804100000001, 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.535 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.142058+0000", 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.142058+0000", 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.142058+0000", 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7599999999999997e-07 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.536 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.142063+0000", 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.87e-06 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.142075+0000", 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.2048000000000001e-05 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.142082+0000", 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.494e-06 2024-04-26T01:25:12.537 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.538 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.538 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.538 INFO:teuthology.orchestra.run.smithi045.stdout: ], 2024-04-26T01:25:12.538 INFO:teuthology.orchestra.run.smithi045.stdout: "num_ops": 11 2024-04-26T01:25:12.538 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:12.539 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:25:12.540 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:12.543+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:12.641 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:12.643+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:12.655 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 3 to smithi18128752-24 2024-04-26T01:25:12.656 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 4 to smithi18128752-24 2024-04-26T01:25:12.656 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 5 to smithi18128752-24 2024-04-26T01:25:12.661 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:12.661 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [], 2024-04-26T01:25:12.661 INFO:teuthology.orchestra.run.smithi045.stdout: "complaint_time": 30, 2024-04-26T01:25:12.661 INFO:teuthology.orchestra.run.smithi045.stdout: "num_blocked_ops": 0 2024-04-26T01:25:12.661 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:12.676 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:25:12.687 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:25:12.760 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 6 to smithi18128752-24 2024-04-26T01:25:12.760 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 24 updating version 0 to 101 2024-04-26T01:25:12.761 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 24 version 101 is already newer than 98 2024-04-26T01:25:12.761 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 24 version 101 is already newer than 99 2024-04-26T01:25:12.761 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 24 version 101 is already newer than 100 2024-04-26T01:25:12.761 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 24 updating version 101 to 102 2024-04-26T01:25:12.761 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 102 (ObjNum 132 snap 33 seq_num 132) dirty exists 2024-04-26T01:25:12.761 INFO:tasks.rados.rados.0.smithi181.stdout:362: left oid 24 (ObjNum 132 snap 33 seq_num 132) 2024-04-26T01:25:12.762 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:12.767+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:12.839 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:25:12.839 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:25:12.840 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-04-26T01:25:12.840 INFO:teuthology.orchestra.run.smithi045.stdout: "size": 20, 2024-04-26T01:25:12.840 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 600, 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "ops": [ 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:129 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 67.269150550000006, 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.91360868200000001, 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 129 2024-04-26T01:25:12.841 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575105+0000", 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.842 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575106+0000", 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.043e-06 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575111+0000", 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.3370000000000001e-06 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575128+0000", 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7600000000000001e-05 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.843 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.575138+0000", 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.6190000000000006e-06 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.877914+0000", 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30277600900000001 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.878073+0000", 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015861299999999999 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.844 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979098+0000", 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.101025446 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:05.979375+0000", 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000277309 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.488714+0000", 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.50933870599999997 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.845 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:142 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~425984,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 66.734614544999999, 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.88158457700000004, 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.846 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 142 2024-04-26T01:25:12.847 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.847 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.847 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.847 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109641+0000", 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.109644+0000", 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.0970000000000002e-06 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.848 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110339+0000", 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00069456900000000002 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110373+0000", 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.358e-05 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.110388+0000", 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5726000000000002e-05 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.849 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181183+0000", 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.070795134999999995 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.181419+0000", 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00023538399999999999 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.182735+0000", 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013160540000000001 2024-04-26T01:25:12.850 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589098+0000", 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40636337 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991098+0000", 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.401999885 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.851 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991226+0000", 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00012777700000000001 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:143 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:25:12.852 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 66.728446882, 2024-04-26T01:25:12.853 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87574223200000001, 2024-04-26T01:25:12.853 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.853 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.853 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 143 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.854 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115809+0000", 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.115811+0000", 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.317e-06 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116338+0000", 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00052673300000000004 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.855 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116375+0000", 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.6922999999999999e-05 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116392+0000", 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6629999999999998e-05 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284095+0000", 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16770365000000001 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.856 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.284265+0000", 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016969199999999999 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.285568+0000", 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001302554 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589101+0000", 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30353306000000002 2024-04-26T01:25:12.857 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991441+0000", 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40234051300000001 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991551+0000", 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00011016 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.858 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:144 3.8s0 3:1fe6984d:::smithi18128752-48:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 66.727836487000005, 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87544766600000001, 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 144 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.859 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116419+0000", 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116421+0000", 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.15e-06 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.860 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116427+0000", 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9070000000000001e-06 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116443+0000", 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.525e-05 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116452+0000", 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.0890000000000006e-06 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.861 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286741+0000", 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.170288786 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.286883+0000", 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000142683 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.287610+0000", 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000727172 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.862 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.589207+0000", 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30159656099999999 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991768+0000", 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40256125399999998 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.991867+0000", 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.8814000000000006e-05 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.863 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:145 3.8s0 3:1fe6984d:::smithi18128752-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 66.727790381000005, 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.98038499800000001, 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 145 2024-04-26T01:25:12.864 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116466+0000", 2024-04-26T01:25:12.865 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5600000000000004e-07 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116470+0000", 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.3100000000000001e-06 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.116484+0000", 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.3837e-05 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.216615+0000", 2024-04-26T01:25:12.866 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.100131365 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992549+0000", 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.77593446499999996 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:06.992720+0000", 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00017066299999999999 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:07.096851+0000", 2024-04-26T01:25:12.867 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.104130502 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:269 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 60.670534216, 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.81822368899999998, 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.868 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 269 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173722+0000", 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.869 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173723+0000", 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4399999999999999e-07 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173738+0000", 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5798999999999999e-05 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173754+0000", 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5943e-05 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.870 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.173763+0000", 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.1730000000000001e-06 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385572+0000", 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.211808092 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.385724+0000", 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015250999999999999 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.871 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589244+0000", 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.203520116 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.589408+0000", 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00016355699999999999 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:12.991945+0000", 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40253765499999999 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.872 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:395 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.603782234999997, 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.89472404000000005, 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 395 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.873 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.874 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240474+0000", 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.4199999999999995e-07 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240479+0000", 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8199999999999996e-06 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.875 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240489+0000", 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7089999999999997e-06 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240496+0000", 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.9920000000000001e-06 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829732+0000", 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.58923568800000004 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.876 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.829882+0000", 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015056100000000001 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.135198+0000", 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.305315528 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.877 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:396 3.8s0 3:1fe6984d:::smithi18128752-48:4 [read 0~802816,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 49.603725894, 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.99608940400000001, 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 396 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.878 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240530+0000", 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240531+0000", 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.0800000000000004e-07 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.879 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240536+0000", 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2059999999999998e-06 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240547+0000", 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.1663e-05 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.240555+0000", 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.1010000000000003e-06 2024-04-26T01:25:12.880 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.831852+0000", 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.59129766500000003 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:23.832019+0000", 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000166773 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:24.236619+0000", 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40460038799999998 2024-04-26T01:25:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:608 3.8s0 3:18914e2a:::smithi18128752-43:head [append 0~409600] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.805377+0000", 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.038879345000002, 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.87196500700000001, 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.882 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 608 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.805377+0000", 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.805377+0000", 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.805379+0000", 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2900000000000001e-06 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806136+0000", 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000757601 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806169+0000", 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2316e-05 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806184+0000", 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5546e-05 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.897941+0000", 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091756901000000002 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.885 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.897953+0000", 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.217e-05 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.933446+0000", 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.035492194999999997 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.933600+0000", 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015479 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.886 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.934868+0000", 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012679200000000001 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.938268+0000", 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0033999490000000002 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.677202+0000", 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.73893376499999996 2024-04-26T01:25:12.887 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.677342+0000", 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00013956400000000001 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:609 3.8s0 3:18914e2a:::smithi18128752-43:head [setxattr _header (57) in=7b] snapc 12=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.806213+0000", 2024-04-26T01:25:12.888 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.038042915999998, 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.97183218800000004, 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 609 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.889 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806213+0000", 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806213+0000", 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806214+0000", 2024-04-26T01:25:12.890 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4300000000000001e-06 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806225+0000", 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0906e-05 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806246+0000", 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0276999999999999e-05 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806261+0000", 2024-04-26T01:25:12.891 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5136e-05 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.898005+0000", 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.091744395000000006 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.898014+0000", 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4460000000000001e-06 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.892 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.936586+0000", 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.038572137999999999 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.936741+0000", 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015529000000000001 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.937477+0000", 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00073645700000000004 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.893 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.940385+0000", 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0029071230000000002 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.777805+0000", 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.83742000500000002 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.778045+0000", 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000240585 2024-04-26T01:25:12.894 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:610 3.8s0 3:18914e2a:::smithi18128752-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e72)", 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:43.806287+0000", 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 29.037969203999999, 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0756521729999999, 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.895 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 610 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806287+0000", 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806287+0000", 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.896 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806288+0000", 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4869999999999999e-06 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806295+0000", 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.4359999999999999e-06 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806320+0000", 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.5343000000000001e-05 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.897 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.806335+0000", 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.4895999999999999e-05 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911536+0000", 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.105201143 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.911557+0000", 2024-04-26T01:25:12.898 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.0933000000000001e-05 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.938102+0000", 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.026545052 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:43.938251+0000", 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00014903099999999999 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.778133+0000", 2024-04-26T01:25:12.899 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.83988231599999996 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.778287+0000", 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015381699999999999 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:44.881939+0000", 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.103651719 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.900 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:762 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~425984] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.138643+0000", 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.705613197000002, 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.80629500200000004, 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 762 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.901 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.138643+0000", 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.138643+0000", 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.138644+0000", 2024-04-26T01:25:12.902 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7740000000000001e-06 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139006+0000", 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00036113999999999997 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139025+0000", 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.897e-05 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.903 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139035+0000", 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0054000000000001e-05 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.232192+0000", 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.093157191 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.232241+0000", 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.9400000000000001e-05 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.904 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.232928+0000", 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00068700300000000005 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.236598+0000", 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0036697919999999998 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.944853+0000", 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.70825485799999999 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.905 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.944938+0000", 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.4820000000000004e-05 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:763 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~606208] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.139052+0000", 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.705203837999999, 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.806199369, 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.906 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 763 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139052+0000", 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.907 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139052+0000", 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139053+0000", 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.9899999999999999e-07 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139924+0000", 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00087108400000000005 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.908 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139940+0000", 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6509000000000001e-05 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139948+0000", 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0299999999999994e-06 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.233851+0000", 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.093902584999999997 2024-04-26T01:25:12.909 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.233956+0000", 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000104473 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.234754+0000", 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00079840599999999999 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.338319+0000", 2024-04-26T01:25:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10356482 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.945148+0000", 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60682875800000002 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.945251+0000", 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000103905 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.911 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:764 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~540672] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.139964+0000", 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.704291712, 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.805611466, 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 764 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.912 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139964+0000", 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139964+0000", 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.139965+0000", 2024-04-26T01:25:12.913 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.3799999999999996e-07 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.140527+0000", 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00056251700000000001 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.140544+0000", 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.6611000000000001e-05 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.914 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.140553+0000", 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.9530000000000005e-06 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.235386+0000", 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.094832564999999994 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.235483+0000", 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.7201000000000007e-05 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.915 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.236197+0000", 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00071380499999999997 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.338567+0000", 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.102370401 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.945423+0000", 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60685562100000001 2024-04-26T01:25:12.916 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.945576+0000", 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00015305399999999999 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:765 3.8s0 3:14c0c421:::smithi18128752-31:head [append 0~409600] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.140570+0000", 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.703686325, 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.80527490599999996, 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.917 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 765 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.140570+0000", 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.918 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.140570+0000", 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.140570+0000", 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 7.2500000000000005e-07 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.140987+0000", 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00041624000000000001 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.919 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141002+0000", 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5322999999999999e-05 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141013+0000", 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0740000000000001e-05 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.334600+0000", 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19358716200000001 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.920 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.334895+0000", 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00029569500000000001 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.336808+0000", 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001912508 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.339699+0000", 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0028906349999999999 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.921 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.945750+0000", 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.60605097399999996 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.945844+0000", 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.4903999999999998e-05 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.922 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:766 3.8s0 3:14c0c421:::smithi18128752-31:head [setxattr _header (57) in=7b] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:24:56.141030+0000", 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 16.703226408999999, 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.80529577799999996, 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 766 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.923 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141030+0000", 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141030+0000", 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141030+0000", 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7599999999999997e-07 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.924 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141035+0000", 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 5.2279999999999998e-06 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141046+0000", 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.0916e-05 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.141053+0000", 2024-04-26T01:25:12.925 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 6.7889999999999997e-06 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.338874+0000", 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.19782082200000001 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.338975+0000", 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000101118 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.926 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.339428+0000", 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00045323100000000001 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.340756+0000", 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0013274269999999999 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.946186+0000", 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.605430684 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.927 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:24:56.946325+0000", 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000138887 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:885 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~622592] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:11.032592+0000", 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.8116637770000001, 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.90406246300000004, 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.928 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 885 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.032592+0000", 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.929 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.032592+0000", 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.032596+0000", 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.5260000000000002e-06 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.033912+0000", 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001316415 2024-04-26T01:25:12.930 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.033947+0000", 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.4532000000000001e-05 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.033964+0000", 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.7893999999999998e-05 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.199363+0000", 2024-04-26T01:25:12.931 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.16539812500000001 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "op must wait for map", 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.199404+0000", 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.1109999999999998e-05 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.305450+0000", 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.106046732 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.932 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.305853+0000", 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000403008 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.309383+0000", 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0035297890000000002 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.316537+0000", 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0071540850000000001 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.933 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.936005+0000", 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.61946800899999999 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.936655+0000", 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00064923800000000001 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:886 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~655360] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:12.934 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:11.039617+0000", 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.8046390839999999, 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.89788232999999995, 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 886 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.039617+0000", 2024-04-26T01:25:12.935 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.039617+0000", 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.039625+0000", 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.0770000000000008e-06 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.040838+0000", 2024-04-26T01:25:12.936 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012134719999999999 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.040871+0000", 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2826e-05 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.040887+0000", 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5517000000000001e-05 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.937 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.199614+0000", 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15872740399999999 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.199663+0000", 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.8646000000000001e-05 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.312381+0000", 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.112718317 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.938 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.312628+0000", 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024709999999999999 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.314614+0000", 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.001986023 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.421282+0000", 2024-04-26T01:25:12.939 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10666790299999999 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.936885+0000", 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.51560246300000001 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.937499+0000", 2024-04-26T01:25:12.940 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00061458200000000002 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:887 3.8s0 3:1fe6984d:::smithi18128752-48:head [append 0~688128] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:11.046791+0000", 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.797464723, 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.89154768699999998, 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:12.941 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 887 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.046791+0000", 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.046791+0000", 2024-04-26T01:25:12.942 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.046793+0000", 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 2.2570000000000002e-06 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.048019+0000", 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0012259020000000001 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.943 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.048052+0000", 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 3.2566e-05 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.148187+0000", 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.10013506799999999 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.300107+0000", 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.15192012799999999 2024-04-26T01:25:12.944 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.300127+0000", 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.9692999999999999e-05 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.416866+0000", 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11673953199999999 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.417150+0000", 2024-04-26T01:25:12.945 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00028334399999999999 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_started", 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.419162+0000", 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.0020126620000000001 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "sub_op_committed", 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.529934+0000", 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.11077150400000001 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "commit_sent", 2024-04-26T01:25:12.946 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.937730+0000", 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.40779660600000001 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.938339+0000", 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.000608425 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: "description": "osd_op(client.4589.0:890 3.8s0 3:1fe6984d:::smithi18128752-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: "initiated_at": "2024-04-26T01:25:11.248619+0000", 2024-04-26T01:25:12.947 INFO:teuthology.orchestra.run.smithi045.stdout: "age": 1.595636584, 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.001142704, 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "continuous": false, 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "type_data": { 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "flag_point": "started", 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "client_info": { 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "client": "client.4589", 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "tid": 890 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "events": [ 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "initiated", 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.248619+0000", 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.948 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "header_read", 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.248619+0000", 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "throttled", 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.248620+0000", 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.9699999999999994e-07 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "all_read", 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.248624+0000", 2024-04-26T01:25:12.949 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 4.0770000000000001e-06 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "dispatched", 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.248640+0000", 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 1.5679999999999999e-05 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "queued_for_pg", 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.248649+0000", 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 8.5860000000000005e-06 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.950 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.528830+0000", 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.280181764 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.529072+0000", 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.00024133999999999999 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "reached_pg", 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.940938+0000", 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.41186605900000001 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.951 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "started", 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:11.941034+0000", 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 9.5994000000000003e-05 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: { 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "event": "done", 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "time": "2024-04-26T01:25:12.249762+0000", 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: "duration": 0.30872820699999998 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.952 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-04-26T01:25:12.953 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-04-26T01:25:12.953 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-04-26T01:25:12.954 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:25:12.954 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:25:12.955 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:25:12.955 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:25:12.956 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:25:12.956 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:25:12.956 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:25:12.957 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:25:12.957 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:25:12.957 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:25:12.958 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:25:12.958 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:12.959+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:25:12.976 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:25:12.999 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:25:12.999 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:25:13.010 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is set 2024-04-26T01:25:13.010 INFO:teuthology.orchestra.run.smithi037.stderr:set pool 3 pg_num to 26 2024-04-26T01:25:13.016 INFO:tasks.rados.rados.0.smithi181.stdout:350: done (4 left) 2024-04-26T01:25:13.016 INFO:tasks.rados.rados.0.smithi181.stdout:356: done (3 left) 2024-04-26T01:25:13.016 INFO:tasks.rados.rados.0.smithi181.stdout:362: done (2 left) 2024-04-26T01:25:13.016 INFO:tasks.rados.rados.0.smithi181.stdout:364: done (1 left) 2024-04-26T01:25:13.016 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 37 current snap is 34 2024-04-26T01:25:13.016 INFO:tasks.rados.rados.0.smithi181.stdout:365: seq_num 133 ranges {2195456=688128,2883584=720896,3604480=475136,4079616=360448} 2024-04-26T01:25:13.021 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:25:13.021 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:25:13.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: writing smithi18128752-37 from 2195456 to 2883584 tid 1 2024-04-26T01:25:13.040 INFO:tasks.rados.rados.0.smithi181.stdout:365: writing smithi18128752-37 from 2883584 to 3604480 tid 2 2024-04-26T01:25:13.043 INFO:tasks.rados.rados.0.smithi181.stdout:365: writing smithi18128752-37 from 3604480 to 4079616 tid 3 2024-04-26T01:25:13.045 INFO:tasks.rados.rados.0.smithi181.stdout:365: writing smithi18128752-37 from 4079616 to 4440064 tid 4 2024-04-26T01:25:13.045 INFO:tasks.rados.rados.0.smithi181.stdout:366: snap_remove snap 33 2024-04-26T01:25:13.047 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:25:13.047 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:25:13.055 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:13.056+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:13.070 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:25:13.071 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:25:13.136 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:13.136 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:13.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:25:13.194 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:25:13.195 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:25:13.198 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:13.203+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:13.217 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:25:13.218 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:25:13.241 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:25:13.241 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:25:13.257 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:13.259+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:13.276 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:13.276 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:13.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:25:13.295 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:13.295 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:13.357 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:13.359+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:13.394 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:13.394 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:13.458 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:13.458 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:13.459 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:13.460+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:13.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:25:13.560 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:13.564+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:13.601 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:13.601 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:13.616 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:25:13.661 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:13.663+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:13.742 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:13.743 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:13.758 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:25:13.762 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:13.762+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:13.864 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:13.868+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:13.877 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:13.878 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:13.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:25:13.964 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:13.968+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:14.028 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:14.028 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing write tid 1 to smithi18128752-37 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing write tid 2 to smithi18128752-37 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing write tid 3 to smithi18128752-37 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing write tid 4 to smithi18128752-37 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing write tid 5 to smithi18128752-37 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing write tid 6 to smithi18128752-37 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: oid 37 updating version 0 to 78 2024-04-26T01:25:14.035 INFO:tasks.rados.rados.0.smithi181.stdout:365: oid 37 version 78 is already newer than 77 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:365: oid 37 updating version 78 to 79 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:365: oid 37 version 79 is already newer than 76 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:365: oid 37 version 79 is already newer than 75 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 79 (ObjNum 133 snap 34 seq_num 133) dirty exists 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:365: left oid 37 (ObjNum 133 snap 34 seq_num 133) 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:365: done (2 left) 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:366: done (1 left) 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:367: read oid 22 snap -1 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:367: expect (ObjNum 108 snap 21 seq_num 108) 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:368: copy_from oid 46 from oid 16 current snap is 34 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 39 current snap is 34 2024-04-26T01:25:14.036 INFO:tasks.rados.rados.0.smithi181.stdout:369: seq_num 134 ranges {0=704512,704512=475136,1179648=573440,1753088=507904} 2024-04-26T01:25:14.038 INFO:tasks.rados.rados.0.smithi181.stdout:369: writing smithi18128752-39 from 0 to 704512 tid 1 2024-04-26T01:25:14.044 INFO:tasks.rados.rados.0.smithi181.stdout:369: writing smithi18128752-39 from 704512 to 1179648 tid 2 2024-04-26T01:25:14.048 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:25:14.048 INFO:tasks.rados.rados.0.smithi181.stdout:369: writing smithi18128752-39 from 1179648 to 1753088 tid 3 2024-04-26T01:25:14.053 INFO:tasks.rados.rados.0.smithi181.stdout:369: writing smithi18128752-39 from 1753088 to 2260992 tid 4 2024-04-26T01:25:14.053 INFO:tasks.rados.rados.0.smithi181.stdout:370: read oid 41 snap -1 2024-04-26T01:25:14.053 INFO:tasks.rados.rados.0.smithi181.stdout:370: expect deleted 2024-04-26T01:25:14.053 INFO:tasks.rados.rados.0.smithi181.stdout:371: rmattr oid 17 current snap is 34 2024-04-26T01:25:14.053 INFO:tasks.rados.rados.0.smithi181.stdout:371: done (5 left) 2024-04-26T01:25:14.053 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 33 current snap is 34 2024-04-26T01:25:14.053 INFO:tasks.rados.rados.0.smithi181.stdout:372: seq_num 135 ranges {1540096=720896,2260992=622592,2883584=688128,3571712=245760} 2024-04-26T01:25:14.065 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:14.067+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:14.069 INFO:tasks.rados.rados.0.smithi181.stdout:372: writing smithi18128752-33 from 1540096 to 2260992 tid 1 2024-04-26T01:25:14.073 INFO:tasks.rados.rados.0.smithi181.stdout:372: writing smithi18128752-33 from 2260992 to 2883584 tid 2 2024-04-26T01:25:14.078 INFO:tasks.rados.rados.0.smithi181.stdout:372: writing smithi18128752-33 from 2883584 to 3571712 tid 3 2024-04-26T01:25:14.081 INFO:tasks.rados.rados.0.smithi181.stdout:372: writing smithi18128752-33 from 3571712 to 3817472 tid 4 2024-04-26T01:25:14.081 INFO:tasks.rados.rados.0.smithi181.stdout:373: read oid 2 snap -1 2024-04-26T01:25:14.081 INFO:tasks.rados.rados.0.smithi181.stdout:373: expect deleted 2024-04-26T01:25:14.081 INFO:tasks.rados.rados.0.smithi181.stdout:374: copy_from oid 24 from oid 44 current snap is 34 2024-04-26T01:25:14.081 INFO:tasks.rados.rados.0.smithi181.stdout:375: read oid 7 snap -1 2024-04-26T01:25:14.081 INFO:tasks.rados.rados.0.smithi181.stdout:375: expect (ObjNum 92 snap 17 seq_num 92) 2024-04-26T01:25:14.082 INFO:tasks.rados.rados.0.smithi181.stdout:376: read oid 28 snap 27 2024-04-26T01:25:14.082 INFO:tasks.rados.rados.0.smithi181.stdout:376: expect (ObjNum 111 snap 23 seq_num 111) 2024-04-26T01:25:14.082 INFO:tasks.rados.rados.0.smithi181.stdout:377: read oid 47 snap -1 2024-04-26T01:25:14.082 INFO:tasks.rados.rados.0.smithi181.stdout:377: expect (ObjNum 97 snap 17 seq_num 97) 2024-04-26T01:25:14.082 INFO:tasks.rados.rados.0.smithi181.stdout:378: copy_from oid 31 from oid 40 current snap is 34 2024-04-26T01:25:14.082 INFO:tasks.rados.rados.0.smithi181.stdout:379: snap_create 2024-04-26T01:25:14.166 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:14.171+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:14.169 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:14.169 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:14.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:25:14.267 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:14.271+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:14.305 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:14.305 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:14.320 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:25:14.340 INFO:tasks.rados.rados.0.smithi181.stdout:367: expect (ObjNum 108 snap 21 seq_num 108) 2024-04-26T01:25:14.368 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:14.371+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:14.438 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:14.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:25:14.469 INFO:tasks.ceph.osd.1.smithi045.stderr:2024-04-26T01:25:14.471+0000 7f76a49e4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 24935) UID: 0 2024-04-26T01:25:14.559 INFO:tasks.rados.rados.0.smithi181.stdout:377: expect (ObjNum 97 snap 17 seq_num 97) 2024-04-26T01:25:14.570 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:14.575+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:14.573 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:14.574 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:14.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:25:14.671 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:14.670+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:14.704 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:14.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:25:14.771 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:14.776+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:14.872 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:14.876+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:14.883 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:14.883 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:14.897 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:25:14.973 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:14.974+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:15.013 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:15.013 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:15.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:25:15.031 INFO:tasks.rados.rados.0.smithi181.stdout:367: done (12 left) 2024-04-26T01:25:15.031 INFO:tasks.rados.rados.0.smithi181.stdout:370: done (11 left) 2024-04-26T01:25:15.031 INFO:tasks.rados.rados.0.smithi181.stdout:377: done (10 left) 2024-04-26T01:25:15.031 INFO:tasks.rados.rados.0.smithi181.stdout:379: done (9 left) 2024-04-26T01:25:15.031 INFO:tasks.rados.rados.0.smithi181.stdout:380: delete oid 41 current snap is 35 2024-04-26T01:25:15.036 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:25:15.086 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:15.091+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:15.175 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:15.179+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:15.194 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:15.194 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:15.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:25:15.276 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:15.274+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:15.332 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:15.332 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:15.340 INFO:tasks.rados.rados.0.smithi181.stdout:380: done (9 left) 2024-04-26T01:25:15.340 INFO:tasks.rados.rados.0.smithi181.stdout:381: delete oid 20 current snap is 35 2024-04-26T01:25:15.347 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:25:15.377 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:15.379+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:15.423 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T01:25:15.477 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:15.478+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:15.507 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:15.508 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:15.522 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:25:15.546 INFO:tasks.rados.rados.0.smithi181.stdout:373: done (9 left) 2024-04-26T01:25:15.547 INFO:tasks.rados.rados.0.smithi181.stdout:381: done (8 left) 2024-04-26T01:25:15.547 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 22 current snap is 35 2024-04-26T01:25:15.547 INFO:tasks.rados.rados.0.smithi181.stdout:382: seq_num 138 ranges {1818624=770048,2588672=540672,3129344=753664,3883008=262144} 2024-04-26T01:25:15.566 INFO:tasks.rados.rados.0.smithi181.stdout:382: writing smithi18128752-22 from 1818624 to 2588672 tid 1 2024-04-26T01:25:15.570 INFO:tasks.rados.rados.0.smithi181.stdout:382: writing smithi18128752-22 from 2588672 to 3129344 tid 2 2024-04-26T01:25:15.576 INFO:tasks.rados.rados.0.smithi181.stdout:382: writing smithi18128752-22 from 3129344 to 3883008 tid 3 2024-04-26T01:25:15.578 INFO:tasks.rados.rados.0.smithi181.stdout:382: writing smithi18128752-22 from 3883008 to 4145152 tid 4 2024-04-26T01:25:15.578 INFO:tasks.rados.rados.0.smithi181.stdout:383: snap_create 2024-04-26T01:25:15.579 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:15.580+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:15.680 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:15.683+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:15.689 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:15.689 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:15.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:25:15.767 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:25:15.767 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:25:15.767 INFO:tasks.rados.rados.0.smithi181.stdout:382: finishing write tid 1 to smithi18128752-22 2024-04-26T01:25:15.781 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:15.782+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:15.793 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:25:15.794 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:25:15.822 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:25:15.823 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:25:15.839 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:15.840 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:15.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:25:15.855 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:25:15.856 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:25:15.868 INFO:tasks.rados.rados.0.smithi181.stdout:374: finishing copy_from racing read to smithi18128752-24 2024-04-26T01:25:15.879 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:25:15.879 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:25:15.881 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:15.883+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:15.902 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:25:15.902 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:25:15.927 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:25:15.928 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:25:15.954 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:25:15.955 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:25:15.959 INFO:tasks.rados.rados.0.smithi181.stdout:369: finishing write tid 1 to smithi18128752-39 2024-04-26T01:25:15.959 INFO:tasks.rados.rados.0.smithi181.stdout:369: finishing write tid 2 to smithi18128752-39 2024-04-26T01:25:15.959 INFO:tasks.rados.rados.0.smithi181.stdout:369: finishing write tid 3 to smithi18128752-39 2024-04-26T01:25:15.959 INFO:tasks.rados.rados.0.smithi181.stdout:369: finishing write tid 4 to smithi18128752-39 2024-04-26T01:25:15.959 INFO:tasks.rados.rados.0.smithi181.stdout:369: finishing write tid 5 to smithi18128752-39 2024-04-26T01:25:15.978 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:25:15.978 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:25:15.982 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:25:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:25:15.983 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:15.983 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:15.982+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:15.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:25:16.002 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:25:16.002 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:25:16.026 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:25:16.026 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:25:16.043 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:25:16.050 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:25:16.051 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:25:16.053 INFO:tasks.rados.rados.0.smithi181.stdout:383: done (9 left) 2024-04-26T01:25:16.053 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 15 current snap is 36 2024-04-26T01:25:16.053 INFO:tasks.rados.rados.0.smithi181.stdout:384: seq_num 139 ranges {1982464=786432,2768896=688128,3457024=475136,3932160=393216} 2024-04-26T01:25:16.076 INFO:tasks.rados.rados.0.smithi181.stdout:384: writing smithi18128752-15 from 1982464 to 2768896 tid 1 2024-04-26T01:25:16.081 INFO:tasks.rados.rados.0.smithi181.stdout:384: writing smithi18128752-15 from 2768896 to 3457024 tid 2 2024-04-26T01:25:16.085 INFO:tasks.rados.rados.0.smithi181.stdout:384: writing smithi18128752-15 from 3457024 to 3932160 tid 3 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:384: writing smithi18128752-15 from 3932160 to 4325376 tid 4 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:385: rollback oid 12 current snap is 36 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 12 to 30 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:374: finishing copy_from to smithi18128752-24 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:374: done (10 left) 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 34 current snap is 36 2024-04-26T01:25:16.089 INFO:tasks.rados.rados.0.smithi181.stdout:386: seq_num 140 ranges {0=802816,802816=425984,1228800=720896,1949696=409600} 2024-04-26T01:25:16.096 INFO:tasks.rados.rados.0.smithi181.stdout:386: writing smithi18128752-34 from 0 to 802816 tid 1 2024-04-26T01:25:16.099 INFO:tasks.rados.rados.0.smithi181.stdout:386: writing smithi18128752-34 from 802816 to 1228800 tid 2 2024-04-26T01:25:16.105 INFO:tasks.rados.rados.0.smithi181.stdout:386: writing smithi18128752-34 from 1228800 to 1949696 tid 3 2024-04-26T01:25:16.108 INFO:tasks.rados.rados.0.smithi181.stdout:386: writing smithi18128752-34 from 1949696 to 2359296 tid 4 2024-04-26T01:25:16.108 INFO:tasks.rados.rados.0.smithi181.stdout:387: rmattr oid 19 current snap is 36 2024-04-26T01:25:16.108 INFO:tasks.rados.rados.0.smithi181.stdout:387: done (11 left) 2024-04-26T01:25:16.109 INFO:tasks.rados.rados.0.smithi181.stdout:388: read oid 38 snap -1 2024-04-26T01:25:16.109 INFO:tasks.rados.rados.0.smithi181.stdout:388: expect (ObjNum 121 snap 30 seq_num 121) 2024-04-26T01:25:16.109 INFO:tasks.rados.rados.0.smithi181.stdout:389: snap_create 2024-04-26T01:25:16.109 INFO:tasks.rados.rados.0.smithi181.stdout:382: finishing write tid 2 to smithi18128752-22 2024-04-26T01:25:16.118 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:16.123+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:16.168 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:16.168 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:25:16.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:25:16.168 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:16.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.168 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:169 3.as0 3:56e8b2fd:::smithi18128752-12:head [copy-from ver 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 68.873148624999999, 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.415445014, 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 169 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297980+0000", 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3359999999999997e-06 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297998+0000", 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8499999999999999e-05 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298041+0000", 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3171999999999999e-05 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398202+0000", 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10016098800000001 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398693+0000", 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00049036800000000003 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398913+0000", 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220051 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.610194+0000", 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.211280998 2024-04-26T01:25:16.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.612042+0000", 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018484860000000001 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713242+0000", 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10119953 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.713420+0000", 2024-04-26T01:25:16.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017858500000000001 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:170 3.as0 3:56e8b2fd:::smithi18128752-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 68.772883843000002, 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.213653854, 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.174 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 170 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398240+0000", 2024-04-26T01:25:16.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398242+0000", 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0530000000000001e-06 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398252+0000", 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4830000000000005e-06 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398271+0000", 2024-04-26T01:25:16.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9230000000000001e-05 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.398282+0000", 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1476e-05 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404421+0000", 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006138592 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:16.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.404504+0000", 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2750999999999994e-05 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611404+0000", 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20690029200000001 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611712+0000", 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00030829799999999998 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.611894+0000", 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018167899999999999 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:212 3.as0 3:56e8b2fd:::smithi18128752-12:head [create,setxattr XNeZS[lOPtVca\\Pa[\\rdfbinb[bcjhtYSpmcafZm (1928) in=40b,setxattr TfflYYqPnps]lo^qSlp___TitNdd`]kaPTVOTcsh (1648) in=40b,setxattr TNb_onpY[j_dkhfmnWtm]Yktg]]RNtbQe[OUTNlX (770) in=40b,setxattr ZcgnkhY\\orTYaZh]PPldj`SNq`p\\cX`k]h\\bZY_Y (700) in=40b,setxattr gPQVnqhpsVhW_kq`QnafXrlRZtZ\\YnRftgQTishR (595) in=40b,setxattr U`XqYi[WhoqnpSPRc\\RhnppZp^qY\\qUfjpqs[r^a (1892) in=40b,setxattr cTmhd`pWgrRf_VfVgOtgeOP^ShP]lsOtpoTj[\\gl (418) in=40b,setxattr aa_taYTfslg`lRRrOkkraN\\XiW^V\\ljl]NqXin`a (314) in=40b,setxattr ibl]`igtgVThY^Qse\\okYYUtTmolX\\W]p]Q[P\\il (1403) in=40b,setxattr \\ZabSO[UUqeZa`hddNh[ZjaRRgUZoQtkn\\QdsX]W (1541) in=40b,setxattr jUkQUPTNbo`rZNmYlUmhb]ojc]aeheoVRfp]bPWf (5052) in=40b,setxattr Ofj`oQ^cR]da^SPsTaXmmPse\\OZ`VSqQ]U`efsZl (419) in=40b,setxattr a[cdkWos\\OrUVPciUaa^_^`lWVTScffiWkd\\[kgf (1261) in=40b,setxattr jXb]\\WhbatdZ[qTTiZO]kPd^`l[kajOYiQ[iqSYc (244) in=40b,setxattr \\im]jQhbanUn\\_^h`jaU]W`ReaZN_je[]kgg (1578) in=36b,setxattr ZbosoRORS^_de]rN^_daOiha\\Q\\eRae^[sQUjXsV (9120) in=40b,setxattr gcn]XpZUkOZr\\pkrVVZpYiR\\alh_PaZa_S[PYcn] (384) in=40b,setxattr \\^qUOaNXSTjtmafhfbo\\gseTTkV`mgnacYq^bkcc (34) in=40b,setxattr iUa^mOQ]TqjVSUU_nXfY]f`NjljiiaYbrm[o]aes (1611) in=40b,setxattr tXaQPQbc^aiV[^reVOPpib^caoWf^XgqilO^_g[k (1016) in=40b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 68.050125194000003, 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.306234695, 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.179 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 212 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.120999+0000", 2024-04-26T01:25:16.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121002+0000", 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.366e-06 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121071+0000", 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9202000000000006e-05 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121106+0000", 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5052999999999998e-05 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.121125+0000", 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8219e-05 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221415+0000", 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029012800000001 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.221427+0000", 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.187e-05 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222099+0000", 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067277899999999998 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.222322+0000", 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000222272 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.224059+0000", 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017374059999999999 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.226116+0000", 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020565679999999999 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.426988+0000", 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20087223500000001 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.427234+0000", 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00024559700000000001 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 40/30 ECSubWriteReply(tid=9, last_complete=40'8, committed=1, applied=1))", 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 64.945370230999998, 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.28191508700000001, 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.185 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225754+0000", 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225756+0000", 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.756e-06 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225765+0000", 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6199999999999994e-06 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.225784+0000", 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8539999999999999e-05 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.325876+0000", 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100092051 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507611+0000", 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.181735758 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:11.507669+0000", 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7361999999999997e-05 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 41/30 ECSubWrite(tid=94, reqid=osd.4.0:94, at_version=41'18, trim_to=0'0, roll_forward_to=39'14))", 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 63.667061652000001, 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202154895, 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504062+0000", 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504063+0000", 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4900000000000001e-07 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504085+0000", 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2342999999999999e-05 2024-04-26T01:25:16.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504093+0000", 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9449999999999993e-06 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.504100+0000", 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2470000000000002e-06 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704414+0000", 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20031389399999999 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.704439+0000", 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.499e-05 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706155+0000", 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017162480000000001 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:12.706217+0000", 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1878999999999995e-05 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s2 52/44 ECSubWrite(tid=213, reqid=osd.13.0:212, at_version=0'0, trim_to=0'0, roll_forward_to=52'22))", 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.813453379000002, 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30091122399999998, 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.193 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357671+0000", 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357672+0000", 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.75e-06 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357708+0000", 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5781999999999997e-05 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.357774+0000", 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6131999999999996e-05 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.457877+0000", 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100102935 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458051+0000", 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017357000000000001 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.458066+0000", 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5526e-05 2024-04-26T01:25:16.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658511+0000", 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20044448300000001 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:24.658582+0000", 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1045999999999997e-05 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.as0 65/63 ECSubRead(tid=24, to_read={3:56e8b2fd:::smithi18128752-12:head=0,4096,0}, subchunks={3:56e8b2fd:::smithi18128752-12:head=[0,1]}, attrs_to_read=))", 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:25:16.197 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.226315468000003, 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20271442200000001, 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944809+0000", 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944842+0000", 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3609000000000003e-05 2024-04-26T01:25:16.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.944872+0000", 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9445e-05 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147126+0000", 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202254668 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147523+0000", 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039669999999999999 2024-04-26T01:25:16.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.as0 65/63 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.225390130999997, 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20196552700000001, 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:25:16.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945734+0000", 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945736+0000", 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945793+0000", 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7462e-05 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945835+0000", 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1705999999999997e-05 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.945854+0000", 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9298999999999999e-05 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147627+0000", 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20177293299999999 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147699+0000", 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2384000000000004e-05 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 66/63 ECSubWriteReply(tid=23, last_complete=65'19, committed=1, applied=1))", 2024-04-26T01:25:16.204 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.224547170999998, 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201296893, 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946577+0000", 2024-04-26T01:25:16.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946579+0000", 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0360000000000001e-06 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946590+0000", 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1583000000000001e-05 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946624+0000", 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3849000000000001e-05 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946648+0000", 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3326999999999999e-05 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147789+0000", 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201141864 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.147874+0000", 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4233999999999993e-05 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.as0 66/63 ECSubReadReply(tid=24, attrs_read=0))", 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.224383895000003, 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30137683399999998, 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.208 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946740+0000", 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946741+0000", 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7599999999999997e-07 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946762+0000", 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0993e-05 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946792+0000", 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9776000000000001e-05 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:37.946814+0000", 2024-04-26T01:25:16.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2889999999999999e-05 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.248046+0000", 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30123198000000001 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.248117+0000", 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0518999999999993e-05 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/63 ECSubWriteReply(tid=38, last_complete=77'33, committed=1, applied=1))", 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.213626+0000", 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.957498437000002, 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26814398900000003, 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213626+0000", 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213626+0000", 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213627+0000", 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8759999999999999e-06 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213637+0000", 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7799999999999995e-06 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.213682+0000", 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4736e-05 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.313774+0000", 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100091633 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.481747+0000", 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.16797363800000001 2024-04-26T01:25:16.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.481770+0000", 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2325999999999999e-05 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.as0 77/63 ECSubWriteReply(tid=39, last_complete=77'33, committed=1, applied=1))", 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.807488+0000", 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.363636192000001, 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20182555899999999, 2024-04-26T01:25:16.215 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807488+0000", 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807488+0000", 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807489+0000", 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3629999999999999e-06 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807496+0000", 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7859999999999999e-06 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807515+0000", 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8839999999999999e-05 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.807525+0000", 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0693000000000001e-05 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.009283+0000", 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20175726799999999 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.009313+0000", 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0608999999999998e-05 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.218 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 78/30 ECSubWrite(tid=222, reqid=client.4589.0:680, at_version=78'57, trim_to=0'0, roll_forward_to=72'53))", 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.064452+0000", 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.106671936000001, 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29624553599999998, 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064452+0000", 2024-04-26T01:25:16.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064452+0000", 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064454+0000", 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9130000000000001e-06 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064618+0000", 2024-04-26T01:25:16.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016357700000000001 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064648+0000", 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0788999999999999e-05 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064673+0000", 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4620000000000001e-05 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.258550+0000", 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.193877522 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.258567+0000", 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6274e-05 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360645+0000", 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10207862199999999 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360698+0000", 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.2219000000000003e-05 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 78/30 ECSubWrite(tid=223, reqid=client.4589.0:681, at_version=78'58, trim_to=0'0, roll_forward_to=72'53))", 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.064720+0000", 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.106403875000002, 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29601289400000003, 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.223 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064720+0000", 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064720+0000", 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064722+0000", 2024-04-26T01:25:16.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6700000000000001e-06 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064889+0000", 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000166773 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064910+0000", 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1373e-05 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064929+0000", 2024-04-26T01:25:16.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9179999999999999e-05 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.259402+0000", 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19447310500000001 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.259416+0000", 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3956e-05 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360704+0000", 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101287821 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360733+0000", 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9016000000000001e-05 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 78/30 ECSubWrite(tid=224, reqid=client.4589.0:682, at_version=78'59, trim_to=0'0, roll_forward_to=72'53))", 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.064977+0000", 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.106146663000001, 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29639301600000001, 2024-04-26T01:25:16.227 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064977+0000", 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064977+0000", 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.064979+0000", 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5400000000000001e-06 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.065003+0000", 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4332e-05 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.065020+0000", 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6507e-05 2024-04-26T01:25:16.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.065038+0000", 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7796e-05 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360313+0000", 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.295275909 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.360330+0000", 2024-04-26T01:25:16.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6192e-05 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.361295+0000", 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00096553199999999996 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.361370+0000", 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5208000000000002e-05 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 78/63 ECSubWrite(tid=38, reqid=client.4589.0:693, at_version=78'62, trim_to=0'0, roll_forward_to=72'60))", 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.234436+0000", 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 25.936687578000001, 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20538144699999999, 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234436+0000", 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234436+0000", 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234438+0000", 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1400000000000001e-06 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234476+0000", 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7993e-05 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.234496+0000", 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0543e-05 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.334633+0000", 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100136776 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.435049+0000", 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100416124 2024-04-26T01:25:16.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.435066+0000", 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6816999999999998e-05 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.439736+0000", 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046703090000000001 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.439818+0000", 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1744999999999997e-05 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:768 3.as0 3:56e8b2fd:::smithi18128752-12:18 [read 0~3293184,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:56.072316+0000", 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.098807885999999, 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25833146299999998, 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:16.236 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 768 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072316+0000", 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072316+0000", 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072320+0000", 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4630000000000001e-06 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072335+0000", 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5664000000000001e-05 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072367+0000", 2024-04-26T01:25:16.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1429e-05 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072389+0000", 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2512999999999999e-05 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072581+0000", 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019207999999999999 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072707+0000", 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012527400000000001 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.330648+0000", 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.25794104000000001 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:769 3.as0 3:56e8b2fd:::smithi18128752-12:18 [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:56.072441+0000", 2024-04-26T01:25:16.240 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.098682577999998, 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26148138500000001, 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 769 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072441+0000", 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072441+0000", 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072443+0000", 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.64e-06 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072456+0000", 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3020000000000001e-05 2024-04-26T01:25:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072476+0000", 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9758999999999998e-05 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072493+0000", 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7176999999999998e-05 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073372+0000", 2024-04-26T01:25:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00087873300000000003 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073456+0000", 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3889999999999998e-05 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.333923+0000", 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.26046716600000003 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:770 3.as0 3:56e8b2fd:::smithi18128752-12:18 [read 0~3293184,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:56.072522+0000", 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 20.098601598999998, 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.36229614300000001, 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 770 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.245 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072522+0000", 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072522+0000", 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072524+0000", 2024-04-26T01:25:16.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4330000000000001e-06 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072540+0000", 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5832999999999999e-05 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072561+0000", 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0973000000000001e-05 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.072577+0000", 2024-04-26T01:25:16.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6116e-05 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073742+0000", 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001165555 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.073857+0000", 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011509 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:56.434819+0000", 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.36096114299999998 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.ds1 100/63 ECSubWrite(tid=63, reqid=osd.7.0:63, at_version=100'82, trim_to=0'0, roll_forward_to=99'80))", 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:12.014701+0000", 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.1564228779999999, 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20161712900000001, 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.014701+0000", 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.014701+0000", 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.014702+0000", 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3699999999999999e-07 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.014786+0000", 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3559000000000004e-05 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.014819+0000", 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3495999999999999e-05 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.014844+0000", 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4706e-05 2024-04-26T01:25:16.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.115134+0000", 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100290164 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.115150+0000", 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5556000000000001e-05 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.216229+0000", 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101079559 2024-04-26T01:25:16.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.216318+0000", 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9152000000000003e-05 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.253 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:16.254 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:25:16.255 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:25:16.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:25:16.256 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:16.256+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:16.355 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:16.358+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:16.430 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:16.431 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:16.431 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-04-26T01:25:16.431 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:16.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:25:16.457 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:16.458+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:16.557 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:16.560+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:16.568 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:16.569 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:16.569 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:25:16.569 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:25:16.569 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:16.574 INFO:tasks.rados.rados.0.smithi181.stdout:369: finishing write tid 6 to smithi18128752-39 2024-04-26T01:25:16.574 INFO:tasks.rados.rados.0.smithi181.stdout:369: oid 39 updating version 0 to 38 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:369: oid 39 updating version 38 to 39 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:369: oid 39 updating version 39 to 42 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:369: oid 39 version 42 is already newer than 41 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:369: oid 39 version 42 is already newer than 40 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 39 v 42 (ObjNum 134 snap 34 seq_num 134) dirty exists 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:369: left oid 39 (ObjNum 134 snap 34 seq_num 134) 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:382: finishing write tid 3 to smithi18128752-22 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:382: finishing write tid 4 to smithi18128752-22 2024-04-26T01:25:16.575 INFO:tasks.rados.rados.0.smithi181.stdout:382: finishing write tid 5 to smithi18128752-22 2024-04-26T01:25:16.584 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:25:16.591 INFO:tasks.rados.rados.0.smithi181.stdout:388: expect (ObjNum 121 snap 30 seq_num 121) 2024-04-26T01:25:16.626 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:25:16.626 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:25:16.659 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:16.662+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:16.663 INFO:tasks.rados.rados.0.smithi181.stdout:375: expect (ObjNum 92 snap 17 seq_num 92) 2024-04-26T01:25:16.714 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:16.714 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:25:16.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:25:16.714 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:16.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.714 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:542 3.ds0 3:b6ed1d05:::smithi18128752-41:head [copy-from ver 70] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.87911304, 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.99677863200000005, 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 542 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839335+0000", 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839338+0000", 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7669999999999999e-06 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839350+0000", 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2109e-05 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839372+0000", 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.162e-05 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839396+0000", 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3898999999999999e-05 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986378+0000", 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14698248899999999 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986395+0000", 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6545000000000002e-05 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.196668+0000", 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.210273194 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.196889+0000", 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022093199999999999 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.626756+0000", 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.42986713199999999 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.635172+0000", 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084160469999999994 2024-04-26T01:25:16.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.835968+0000", 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20079548799999999 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.836114+0000", 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014641 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:543 3.ds0 3:b6ed1d05:::smithi18128752-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 37.879013332, 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.79564045900000002, 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 543 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839435+0000", 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839437+0000", 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6130000000000001e-06 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839444+0000", 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7700000000000004e-06 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839465+0000", 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0970000000000001e-05 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.839486+0000", 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1199000000000002e-05 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986531+0000", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14704526000000001 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.986557+0000", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5701999999999999e-05 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197379+0000", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.21082251799999999 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197562+0000", 2024-04-26T01:25:16.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000182652 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.634726+0000", 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43716451899999997 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.634892+0000", 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016576000000000001 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.635076+0000", 2024-04-26T01:25:16.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018349600000000001 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(osd.14.21:4 3.ds0 3:b136579b:::smithi18128752-36:head [assert-version v14,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.925621+0000", 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.792827491000001, 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81766490700000005, 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925621+0000", 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925621+0000", 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925624+0000", 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7159999999999999e-06 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925638+0000", 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.447e-05 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925677+0000", 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9135999999999999e-05 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925692+0000", 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4902999999999999e-05 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925796+0000", 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000103651 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.925804+0000", 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4950000000000008e-06 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.133516+0000", 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20771204700000001 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.133724+0000", 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020719900000000001 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743286+0000", 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60956228999999995 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.730 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:693 3.ds0 3:b6ed1d05:::smithi18128752-41:head [append 0~0] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.988574+0000", 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.729874236000001, 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55445387800000001, 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 693 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988574+0000", 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988574+0000", 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988578+0000", 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8530000000000002e-06 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988594+0000", 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.624e-05 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988631+0000", 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6519000000000001e-05 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988651+0000", 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0233000000000001e-05 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.134541+0000", 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14588947799999999 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.134724+0000", 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018301299999999999 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.135949+0000", 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012256700000000001 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.339321+0000", 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.203372152 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.542894+0000", 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20357222899999999 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543028+0000", 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013449100000000001 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:694 3.ds0 3:b6ed1d05:::smithi18128752-41:head [rollback 13] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.988685+0000", 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.729763600999998, 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55479098800000004, 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 694 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988685+0000", 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988685+0000", 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988686+0000", 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0359999999999999e-06 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988691+0000", 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8799999999999999e-06 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988702+0000", 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1036000000000001e-05 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988711+0000", 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9830000000000002e-06 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.738 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.237077+0000", 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.24836664 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.237256+0000", 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017854599999999999 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.238335+0000", 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00107851 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.739 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.339397+0000", 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101062768 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543374+0000", 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20397695099999999 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.543476+0000", 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010163800000000001 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.740 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:695 3.ds0 3:b6ed1d05:::smithi18128752-41:head [append 0~0] snapc 16=[15,13] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:49.988724+0000", 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.729724799, 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.75505351200000004, 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 695 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.741 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988724+0000", 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988724+0000", 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988725+0000", 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5000000000000001e-07 2024-04-26T01:25:16.742 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988740+0000", 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5573000000000002e-05 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988750+0000", 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0183000000000001e-05 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.743 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:49.988758+0000", 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.14e-06 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.239213+0000", 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.250454385 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.239440+0000", 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022750900000000001 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.240667+0000", 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001226783 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.339400+0000", 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098732746999999996 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743591+0000", 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40419128900000001 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.745 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743777+0000", 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018605299999999999 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.ds0 78/63 ECSubReadReply(tid=37, attrs_read=0))", 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.335466+0000", 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 26.382982420000001, 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40784600500000001, 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.746 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335466+0000", 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335466+0000", 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.747 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335468+0000", 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.437e-06 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335499+0000", 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1173000000000001e-05 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.335534+0000", 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5451999999999999e-05 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.748 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.435652+0000", 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10011783 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.742921+0000", 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30726856800000002 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.743312+0000", 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00039154499999999998 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.749 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:900 3.ds0 3:b136579b:::smithi18128752-36:head [append 0~425984] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.079707+0000", 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6387418040000004, 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.41464551300000002, 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 900 2024-04-26T01:25:16.750 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.079707+0000", 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.079707+0000", 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.751 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.079709+0000", 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5830000000000001e-06 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.080306+0000", 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00059669900000000001 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.080341+0000", 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4755999999999997e-05 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.752 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.080365+0000", 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3830000000000001e-05 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.080494+0000", 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129071 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.080505+0000", 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1199999999999999e-05 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.753 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.086404+0000", 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058994479999999998 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.086537+0000", 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013284700000000001 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.087552+0000", 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001014746 2024-04-26T01:25:16.754 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.292585+0000", 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20503348699999999 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.494228+0000", 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20164242399999999 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.494352+0000", 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012442200000000001 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.755 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:901 3.ds0 3:b136579b:::smithi18128752-36:head [append 0~638976] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.084990+0000", 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6334586890000002, 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.51033795299999996, 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 901 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.756 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.084990+0000", 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.084990+0000", 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.084993+0000", 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1360000000000001e-06 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.757 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.085946+0000", 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000952837 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.085986+0000", 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0312000000000003e-05 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.086010+0000", 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4117e-05 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.758 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.188360+0000", 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102350019 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.188540+0000", 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018011700000000001 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.190539+0000", 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019985020000000001 2024-04-26T01:25:16.759 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.292867+0000", 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102328338 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.595195+0000", 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30232773000000002 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.595328+0000", 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013284500000000001 2024-04-26T01:25:16.760 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:902 3.ds0 3:b136579b:::smithi18128752-36:head [append 0~491520] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.088991+0000", 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6294578670000002, 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.71438587600000003, 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.761 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 902 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.088991+0000", 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.088991+0000", 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.762 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.088994+0000", 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9270000000000001e-06 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089784+0000", 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00079053800000000003 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089823+0000", 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8816000000000002e-05 2024-04-26T01:25:16.763 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089849+0000", 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6529999999999998e-05 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.596188+0000", 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50633837699999995 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.596341+0000", 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015323499999999999 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.764 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.598012+0000", 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016709489999999999 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.701531+0000", 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103518628 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.803179+0000", 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101648746 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.765 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.803377+0000", 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019713 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:903 3.ds0 3:b136579b:::smithi18128752-36:head [setxattr _header (57) in=7b] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.089890+0000", 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6285588129999997, 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.714002778, 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 903 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089890+0000", 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089890+0000", 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.767 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089892+0000", 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0329999999999998e-06 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089901+0000", 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9870000000000005e-06 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089924+0000", 2024-04-26T01:25:16.768 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3753000000000001e-05 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089944+0000", 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9972000000000001e-05 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.699475+0000", 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.609530774 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.699656+0000", 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000180705 2024-04-26T01:25:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.700667+0000", 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010113450000000001 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.702352+0000", 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016843959999999999 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:16.770 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.803738+0000", 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101385962 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.803892+0000", 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015485100000000001 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:904 3.ds0 3:b136579b:::smithi18128752-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.089969+0000", 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.6284795409999999, 2024-04-26T01:25:16.771 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.91866110700000003, 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 904 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089969+0000", 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.772 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089969+0000", 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089970+0000", 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3030000000000001e-06 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089976+0000", 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.964e-06 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.773 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.089999+0000", 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2379e-05 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.190103+0000", 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010416900000001 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.806043+0000", 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61593993000000002 2024-04-26T01:25:16.774 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.806215+0000", 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000172681 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.008630+0000", 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20241468100000001 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.775 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 99/63 ECSubWriteReply(tid=56, last_complete=99'77, committed=1, applied=1))", 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.188097+0000", 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5303512809999997, 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30595635399999999, 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.188097+0000", 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.776 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.188097+0000", 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.188098+0000", 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.167e-06 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.188105+0000", 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.139e-06 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.777 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.188131+0000", 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6679999999999999e-05 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.288276+0000", 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100145003 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.494021+0000", 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20574490500000001 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.778 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.494054+0000", 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2459999999999998e-05 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 99/63 ECSubWriteReply(tid=57, last_complete=99'78, committed=1, applied=1))", 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.193446+0000", 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.5250024849999999, 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40129875100000001, 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.779 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.193446+0000", 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.193446+0000", 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.193448+0000", 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7999999999999999e-06 2024-04-26T01:25:16.780 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.193456+0000", 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9200000000000004e-06 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.193478+0000", 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2123000000000001e-05 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.193491+0000", 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3057000000000001e-05 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.594709+0000", 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40121850100000001 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.594745+0000", 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5349999999999999e-05 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.782 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 99/63 ECSubWriteReply(tid=57, last_complete=99'78, committed=1, applied=1))", 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.290373+0000", 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.4280758279999999, 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30575969400000003, 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290373+0000", 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.783 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290373+0000", 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290375+0000", 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.843e-06 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290384+0000", 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2399999999999996e-06 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290412+0000", 2024-04-26T01:25:16.784 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7943e-05 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290426+0000", 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4316e-05 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.595065+0000", 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30463920300000003 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.596132+0000", 2024-04-26T01:25:16.785 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010671490000000001 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 99/63 ECSubWriteReply(tid=57, last_complete=99'78, committed=1, applied=1))", 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.290510+0000", 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.4279389370000004, 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.304312531, 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.786 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290510+0000", 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290510+0000", 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290511+0000", 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1699999999999997e-07 2024-04-26T01:25:16.787 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290520+0000", 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3500000000000003e-06 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290537+0000", 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7410000000000001e-05 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290547+0000", 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0073e-05 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.788 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.594797+0000", 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30425012200000001 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.594822+0000", 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4658999999999999e-05 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 99/63 ECSubWriteReply(tid=57, last_complete=99'78, committed=1, applied=1))", 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.290940+0000", 2024-04-26T01:25:16.789 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.4275081859999998, 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30407955800000003, 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290940+0000", 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290940+0000", 2024-04-26T01:25:16.790 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290942+0000", 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2899999999999999e-06 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290948+0000", 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3029999999999996e-06 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290965+0000", 2024-04-26T01:25:16.791 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7370000000000001e-05 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.290975+0000", 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4599999999999992e-06 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.594996+0000", 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30402110799999998 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.595020+0000", 2024-04-26T01:25:16.792 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4026999999999999e-05 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 99/63 ECSubWriteReply(tid=57, last_complete=99'78, committed=1, applied=1))", 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.293316+0000", 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.4251326740000003, 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301632293, 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.793 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.293316+0000", 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.293316+0000", 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.293318+0000", 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.697e-06 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.794 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.293326+0000", 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8829999999999995e-06 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.293361+0000", 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4378999999999997e-05 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.293378+0000", 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7173999999999999e-05 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.795 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.594912+0000", 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30153423699999998 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.594948+0000", 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5923000000000001e-05 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 99/63 ECSubWriteReply(tid=63, last_complete=100'82, committed=1, applied=1))", 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:12.020367+0000", 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.6980816110000001, 2024-04-26T01:25:16.796 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29682784099999998, 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.020367+0000", 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.020367+0000", 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.797 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.020369+0000", 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0059999999999999e-06 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.020381+0000", 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.183e-05 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.020413+0000", 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2113999999999998e-05 2024-04-26T01:25:16.798 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.020434+0000", 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0812e-05 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.317158+0000", 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29672403000000003 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.317195+0000", 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7048999999999997e-05 2024-04-26T01:25:16.799 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 100/63 ECSubWriteReply(tid=63, last_complete=100'82, committed=1, applied=1))", 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:12.021713+0000", 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.6967355189999997, 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29717212399999998, 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.800 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.021713+0000", 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.021713+0000", 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.021714+0000", 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.393e-06 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.801 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.021722+0000", 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7130000000000005e-06 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.021743+0000", 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0684000000000001e-05 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.121830+0000", 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100087566 2024-04-26T01:25:16.802 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.318824+0000", 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19699372200000001 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.318885+0000", 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1045999999999998e-05 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.803 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.804 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:16.804 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:16.805 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:16.805 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:16.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:25:16.807 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:16.807 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:16.807+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:372: finishing write tid 1 to smithi18128752-33 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:372: finishing write tid 2 to smithi18128752-33 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:372: finishing write tid 3 to smithi18128752-33 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:372: finishing write tid 4 to smithi18128752-33 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:385: finishing rollback tid 0 to smithi18128752-12 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:385: finishing rollback tid 1 to smithi18128752-12 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:372: finishing write tid 5 to smithi18128752-33 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:382: finishing write tid 6 to smithi18128752-22 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:382: oid 22 updating version 0 to 83 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:382: oid 22 version 83 is already newer than 82 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:382: oid 22 updating version 83 to 85 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:382: oid 22 version 85 is already newer than 84 2024-04-26T01:25:16.842 INFO:tasks.rados.rados.0.smithi181.stdout:382: oid 22 version 85 is already newer than 81 2024-04-26T01:25:16.843 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 85 (ObjNum 138 snap 35 seq_num 138) dirty exists 2024-04-26T01:25:16.843 INFO:tasks.rados.rados.0.smithi181.stdout:382: left oid 22 (ObjNum 138 snap 35 seq_num 138) 2024-04-26T01:25:16.843 INFO:tasks.rados.rados.0.smithi181.stdout:385: finishing rollback tid 2 to smithi18128752-12 2024-04-26T01:25:16.843 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 43 (ObjNum 106 snap 21 seq_num 106) dirty exists 2024-04-26T01:25:16.843 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing copy_from racing read to smithi18128752-31 2024-04-26T01:25:16.843 INFO:tasks.rados.rados.0.smithi181.stdout:384: finishing write tid 1 to smithi18128752-15 2024-04-26T01:25:16.861 INFO:tasks.rados.rados.0.smithi181.stdout:368: finishing copy_from racing read to smithi18128752-46 2024-04-26T01:25:16.905 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:16.906+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:16.909 INFO:tasks.rados.rados.0.smithi181.stdout:384: finishing write tid 2 to smithi18128752-15 2024-04-26T01:25:16.927 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:16.927 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:16.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.927 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.11s1 104/102 ECSubWrite(tid=349, reqid=client.4589.0:925, at_version=104'67, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T01:25:16.927 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:16.860808+0000", 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.072468132000000005, 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072513500999999994, 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "no flag points reached", 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:16.860808+0000", 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:16.860808+0000", 2024-04-26T01:25:16.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:16.860814+0000", 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.536e-06 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:16.866875+0000", 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0060616589999999996 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:16.866896+0000", 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0256999999999999e-05 2024-04-26T01:25:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.930 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:16.930 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.930 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:16.930 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-26T01:25:16.930 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 1 2024-04-26T01:25:16.930 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:16.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:25:16.973 INFO:tasks.rados.rados.0.smithi181.stdout:368: finishing copy_from to smithi18128752-46 2024-04-26T01:25:16.973 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 46 v 60 (ObjNum 120 snap 29 seq_num 120) dirty exists 2024-04-26T01:25:16.986 INFO:tasks.rados.rados.0.smithi181.stdout:376: expect (ObjNum 111 snap 23 seq_num 111) 2024-04-26T01:25:17.005 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:17.008+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:17.094 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:17.095 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:17.095 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:25:17.095 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:25:17.095 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:17.106 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:17.111+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:17.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: finishing write tid 6 to smithi18128752-33 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: oid 33 updating version 0 to 82 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: oid 33 updating version 82 to 85 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: oid 33 version 85 is already newer than 84 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: oid 33 version 85 is already newer than 81 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: oid 33 version 85 is already newer than 83 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 85 (ObjNum 135 snap 34 seq_num 135) dirty exists 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: left oid 33 (ObjNum 135 snap 34 seq_num 135) 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing copy_from to smithi18128752-31 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 163 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:368: done (12 left) 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:369: done (11 left) 2024-04-26T01:25:17.135 INFO:tasks.rados.rados.0.smithi181.stdout:372: done (10 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:375: done (9 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:376: done (8 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:378: done (7 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:382: done (6 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:385: done (5 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:388: done (4 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:389: done (3 left) 2024-04-26T01:25:17.136 INFO:tasks.rados.rados.0.smithi181.stdout:390: delete oid 39 current snap is 37 2024-04-26T01:25:17.207 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:17.212+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:17.259 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:25:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:25:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:150 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~458752,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 71.084438598999995, 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3959368480000001, 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 150 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179061+0000", 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179064+0000", 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7269999999999999e-06 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179608+0000", 2024-04-26T01:25:17.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00054408799999999997 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179641+0000", 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2873000000000002e-05 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.179658+0000", 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6863000000000001e-05 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.262 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268330+0000", 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0886726819999999 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.268562+0000", 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00023123100000000001 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.270316+0000", 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017541189999999999 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.263 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474438+0000", 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20412262 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574871+0000", 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100432788 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.574998+0000", 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000126857 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.264 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:151 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 71.078063736999994, 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.494027684, 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 151 2024-04-26T01:25:17.265 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185436+0000", 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.266 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185438+0000", 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2170000000000001e-06 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185895+0000", 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000456291 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185927+0000", 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1989000000000002e-05 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185942+0000", 2024-04-26T01:25:17.267 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5075999999999999e-05 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272106+0000", 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0861648319999999 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.272243+0000", 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013630399999999999 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.273909+0000", 2024-04-26T01:25:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016665930000000001 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474512+0000", 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20060244399999999 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679353+0000", 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20484124000000001 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679464+0000", 2024-04-26T01:25:17.269 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110698 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:152 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 71.077531831000002, 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4937803999999999, 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.270 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 152 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185968+0000", 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.271 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185969+0000", 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.167e-06 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185975+0000", 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5029999999999997e-06 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185986+0000", 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1283e-05 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.272 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.185994+0000", 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6060000000000003e-06 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372230+0000", 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.186235766 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.372376+0000", 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014574 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.273 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.373451+0000", 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010752890000000001 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.474514+0000", 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101062945 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679632+0000", 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20511794799999999 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.274 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.679748+0000", 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116153 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:153 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 71.077493454999995, 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.900776316, 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.275 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 153 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.276 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186006+0000", 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186007+0000", 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6700000000000002e-07 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186011+0000", 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5559999999999999e-06 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186025+0000", 2024-04-26T01:25:17.277 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3873000000000001e-05 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:06.186035+0000", 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0526000000000001e-05 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475079+0000", 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2890438200000001 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:17.278 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475255+0000", 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000176279 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680464+0000", 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20520917499999999 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.680595+0000", 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013023099999999999 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086783+0000", 2024-04-26T01:25:17.279 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40618798900000003 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:164 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.965586993000002, 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78562828299999998, 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.280 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 164 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297913+0000", 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.281 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297916+0000", 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297932+0000", 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297975+0000", 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2542e-05 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.282 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.297993+0000", 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7813e-05 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475329+0000", 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17733589199999999 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.475446+0000", 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000117074 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.283 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.083541+0000", 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60809543499999996 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:165 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.965471578999995, 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78769006399999997, 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.284 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 165 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298028+0000", 2024-04-26T01:25:17.285 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298030+0000", 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.59e-06 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298039+0000", 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1470000000000007e-06 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.286 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298064+0000", 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5542e-05 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298080+0000", 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5279999999999999e-05 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476021+0000", 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.177941503 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.287 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476177+0000", 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000155873 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.085718+0000", 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60954112900000001 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:166 3.cs0 3:3a8fbb99:::smithi18128752-26:head [read 0~425984,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:25:17.288 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.965395537999996, 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.78831037500000001, 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 166 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:25:17.289 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298104+0000", 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298106+0000", 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-06 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.290 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298113+0000", 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7570000000000004e-06 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298128+0000", 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5e-05 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.298139+0000", 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0986e-05 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.291 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476693+0000", 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17855336099999999 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:07.476835+0000", 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014191599999999999 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086415+0000", 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.60957998499999999 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.292 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:189 3.cs0 3:395c8678:::smithi18128752-25:head [copy-from ver 2] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e37)", 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.186459925999998, 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61527076999999997, 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 189 2024-04-26T01:25:17.293 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077040+0000", 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077044+0000", 2024-04-26T01:25:17.294 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0570000000000003e-06 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077061+0000", 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6616000000000002e-05 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077097+0000", 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6829000000000003e-05 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.295 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.077120+0000", 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2779999999999999e-05 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.086851+0000", 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097310599999999997 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.087007+0000", 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015570199999999999 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.495435+0000", 2024-04-26T01:25:17.296 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.40842789299999999 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.497946+0000", 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025112519999999998 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692090+0000", 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19414421300000001 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:08.692311+0000", 2024-04-26T01:25:17.297 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000220368 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:322 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.113678757000002, 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.707643833, 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.298 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 322 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149821+0000", 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.299 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149822+0000", 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.004e-06 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149829+0000", 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.613e-06 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.149840+0000", 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1819e-05 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.300 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249944+0000", 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100103704 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250099+0000", 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015437400000000001 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250119+0000", 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0392999999999999e-05 2024-04-26T01:25:17.301 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352060+0000", 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101941083 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.352219+0000", 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015886700000000001 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.857465+0000", 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50524597599999999 2024-04-26T01:25:17.302 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:323 3.fs0 3:fccfe559:::smithi18128752-35:4 [read 0~573440,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.013516946999999, 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.70925194000000003, 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.303 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 323 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249983+0000", 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.304 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249984+0000", 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4559999999999999e-06 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.249994+0000", 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0186e-05 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250012+0000", 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7836999999999999e-05 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.305 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.250023+0000", 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1143e-05 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254128+0000", 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041046069999999997 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.254145+0000", 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7405999999999999e-05 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.306 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452144+0000", 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.197998339 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.452432+0000", 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028853800000000002 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.959235+0000", 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.50680242799999997 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.307 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:872 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [append 0~475136] snapc 21=[21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:10.782228+0000", 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.481271338, 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.65969031099999997, 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 872 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.308 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.782228+0000", 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.782228+0000", 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.782230+0000", 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9700000000000002e-06 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.309 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.782899+0000", 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00066847399999999996 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.782932+0000", 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3312000000000002e-05 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.782953+0000", 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1194000000000001e-05 2024-04-26T01:25:17.310 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.786172+0000", 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032190790000000001 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.786332+0000", 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000159932 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.787697+0000", 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00136465 2024-04-26T01:25:17.311 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.891587+0000", 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103889817 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.441797+0000", 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.550209964 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.441919+0000", 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121919 2024-04-26T01:25:17.312 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:873 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [setxattr _header (57) in=7b] snapc 21=[21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:10.783013+0000", 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.4804870589999997, 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.65928352599999995, 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.313 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 873 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783013+0000", 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783013+0000", 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.314 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783015+0000", 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8899999999999999e-06 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783027+0000", 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2799999999999999e-05 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783055+0000", 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7662e-05 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.315 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783080+0000", 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.497e-05 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.788966+0000", 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005885599 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.789159+0000", 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019310400000000001 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.316 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.789981+0000", 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000822411 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.891588+0000", 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101606723 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.442166+0000", 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.55057854900000003 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.317 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.442296+0000", 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000129818 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:874 3.cs0 3:3a0c1dcc:::smithi18128752-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:10.783106+0000", 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.4803938810000004, 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4742328950000001, 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.318 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 874 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783106+0000", 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783106+0000", 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.319 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783107+0000", 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5269999999999999e-06 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783114+0000", 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7429999999999996e-06 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783135+0000", 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1086000000000001e-05 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.783151+0000", 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6154000000000002e-05 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.891126+0000", 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.107975087 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:10.891403+0000", 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027609599999999998 2024-04-26T01:25:17.321 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.443021+0000", 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.551618108 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.443161+0000", 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013999499999999999 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.257339+0000", 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81417809900000004 2024-04-26T01:25:17.322 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:908 3.cs0 3:3c2c778f:::smithi18128752-24:head [append 0~573440] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.116120+0000", 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.147380031, 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.73848894700000001, 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.323 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 908 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.116120+0000", 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.116120+0000", 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.324 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.116123+0000", 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3969999999999998e-06 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.116423+0000", 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029966900000000002 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.116453+0000", 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.021e-05 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.325 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.116468+0000", 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4619e-05 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.549440+0000", 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43297258900000002 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.549598+0000", 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015739199999999999 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.326 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.551077+0000", 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001479572 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.753897+0000", 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20282011699999999 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.854486+0000", 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100588444 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.327 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.854609+0000", 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012293800000000001 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:909 3.cs0 3:3c2c778f:::smithi18128752-24:head [append 0~753664] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.120612+0000", 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1428879380000003, 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5408887849999999, 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.328 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 909 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.120612+0000", 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.120612+0000", 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.329 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.120615+0000", 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0130000000000002e-06 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.121363+0000", 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00074805300000000004 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.121393+0000", 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0530000000000001e-05 2024-04-26T01:25:17.330 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.121408+0000", 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4763000000000001e-05 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.257503+0000", 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.136094945 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.257731+0000", 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000227753 2024-04-26T01:25:17.331 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.260139+0000", 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024078010000000002 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.266837+0000", 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066983779999999996 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.661309+0000", 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39447194400000002 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.332 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.661501+0000", 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00019160499999999999 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:910 3.cs0 3:3c2c778f:::smithi18128752-24:head [append 0~229376] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.121908+0000", 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1415913069999997, 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5401990159999999, 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.333 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 910 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.121908+0000", 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.121908+0000", 2024-04-26T01:25:17.334 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.121911+0000", 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2529999999999999e-06 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122191+0000", 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00028017 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122213+0000", 2024-04-26T01:25:17.335 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.226e-05 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122225+0000", 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1806e-05 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.261997+0000", 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.139771997 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.262297+0000", 2024-04-26T01:25:17.336 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00029979300000000002 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.263713+0000", 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001415821 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.266841+0000", 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003128245 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.661985+0000", 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39514369599999999 2024-04-26T01:25:17.337 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.662107+0000", 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012297499999999999 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:911 3.cs0 3:3c2c778f:::smithi18128752-24:head [setxattr _header (57) in=7b] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.122245+0000", 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1412551710000001, 2024-04-26T01:25:17.338 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5403959459999998, 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 911 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122245+0000", 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.339 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122245+0000", 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122246+0000", 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.159e-06 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122252+0000", 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7799999999999997e-06 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.340 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122261+0000", 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8600000000000005e-06 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122274+0000", 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.225e-05 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.264648+0000", 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1423741330000001 2024-04-26T01:25:17.341 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.264800+0000", 2024-04-26T01:25:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015218799999999999 2024-04-26T01:25:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.265588+0000", 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00078795199999999997 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.266845+0000", 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001257123 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.662368+0000", 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39552269099999998 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.343 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.662641+0000", 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027281000000000001 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4589.0:912 3.cs0 3:3c2c778f:::smithi18128752-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.122285+0000", 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1412148420000001, 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6443940769999998, 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-04-26T01:25:17.344 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4589", 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 912 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122285+0000", 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122285+0000", 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.345 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122286+0000", 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5000000000000002e-07 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122289+0000", 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4829999999999999e-06 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122299+0000", 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0304e-05 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.346 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.122307+0000", 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9130000000000003e-06 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.266208+0000", 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.143900825 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.266431+0000", 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022266699999999999 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.663388+0000", 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.39695738800000002 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.663542+0000", 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015355800000000001 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.766679+0000", 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103137189 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.348 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 99/90 ECSubReadReply(tid=152, attrs_read=0))", 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.444172+0000", 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.8193278160000004, 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81265177700000002, 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444172+0000", 2024-04-26T01:25:17.349 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444172+0000", 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444174+0000", 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7430000000000001e-06 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444199+0000", 2024-04-26T01:25:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4963000000000001e-05 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444225+0000", 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6407e-05 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444259+0000", 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3562000000000002e-05 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.256748+0000", 2024-04-26T01:25:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.81248918699999995 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.256824+0000", 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5914999999999994e-05 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpReadReply(3.cs0 99/90 ECSubReadReply(tid=152, attrs_read=0))", 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.444409+0000", 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.8190909160000004, 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.71212218900000002, 2024-04-26T01:25:17.352 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444409+0000", 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444409+0000", 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.353 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444410+0000", 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8999999999999995e-07 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444437+0000", 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7093e-05 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.444478+0000", 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1362000000000001e-05 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.354 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.544608+0000", 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100130098 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.156460+0000", 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.61185213999999999 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.156531+0000", 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0605999999999995e-05 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.355 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.356 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.356 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.356 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:17.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:25:17.358 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:17.359+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:17.429 INFO:tasks.rados.rados.0.smithi181.stdout:384: finishing write tid 3 to smithi18128752-15 2024-04-26T01:25:17.429 INFO:tasks.rados.rados.0.smithi181.stdout:384: finishing write tid 4 to smithi18128752-15 2024-04-26T01:25:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:386: finishing write tid 1 to smithi18128752-34 2024-04-26T01:25:17.458 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:17.459+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:17.529 INFO:tasks.rados.rados.0.smithi181.stdout:390: done (3 left) 2024-04-26T01:25:17.529 INFO:tasks.rados.rados.0.smithi181.stdout:391: rollback oid 45 current snap is 37 2024-04-26T01:25:17.529 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 45 to 36 2024-04-26T01:25:17.529 INFO:tasks.rados.rados.0.smithi181.stdout:392: snap_create 2024-04-26T01:25:17.530 INFO:tasks.rados.rados.0.smithi181.stdout:384: finishing write tid 5 to smithi18128752-15 2024-04-26T01:25:17.531 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.16s3 105/102 ECSubWrite(tid=65, reqid=client.4589.0:987, at_version=0'0, trim_to=0'0, roll_forward_to=105'44))", 2024-04-26T01:25:17.531 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:17.535823+0000", 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.001355403, 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001395152, 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.535823+0000", 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.535823+0000", 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.535826+0000", 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3829999999999999e-06 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.535856+0000", 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9966000000000001e-05 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.535895+0000", 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9010000000000001e-05 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.535919+0000", 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4275999999999999e-05 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 105/103 ECSubWrite(tid=221, reqid=client.4589.0:972, at_version=0'0, trim_to=0'0, roll_forward_to=104'143))", 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:17.536781+0000", 2024-04-26T01:25:17.534 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.00039753800000000001, 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052411000000000005, 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536781+0000", 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536781+0000", 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536783+0000", 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.843e-06 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536814+0000", 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0846000000000002e-05 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536842+0000", 2024-04-26T01:25:17.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8343e-05 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536857+0000", 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.522e-05 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536964+0000", 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010683600000000001 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:17.536979+0000", 2024-04-26T01:25:17.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5218999999999999e-05 2024-04-26T01:25:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-04-26T01:25:17.538 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 2 2024-04-26T01:25:17.538 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:17.538 INFO:tasks.rados.rados.0.smithi181.stdout:386: finishing write tid 2 to smithi18128752-34 2024-04-26T01:25:17.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:25:17.558 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:17.562+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:17.659 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:17.663+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:17.714 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:17.714 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-04-26T01:25:17.714 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-04-26T01:25:17.714 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-04-26T01:25:17.714 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:17.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:25:17.735 INFO:tasks.rados.rados.0.smithi181.stdout:386: finishing write tid 3 to smithi18128752-34 2024-04-26T01:25:17.736 INFO:tasks.rados.rados.0.smithi181.stdout:386: finishing write tid 4 to smithi18128752-34 2024-04-26T01:25:17.760 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:17.763+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:17.833 INFO:tasks.rados.rados.0.smithi181.stdout:391: finishing rollback tid 0 to smithi18128752-45 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s4 47/30 ECSubWrite(tid=74, reqid=client.4589.0:332, at_version=47'15, trim_to=0'0, roll_forward_to=41'13))", 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 59.282264847999997, 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20225293799999999, 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.853 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575168+0000", 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575170+0000", 2024-04-26T01:25:17.854 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.08e-06 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575220+0000", 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0246e-05 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.575252+0000", 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1482000000000001e-05 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.675393+0000", 2024-04-26T01:25:17.855 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10014118599999999 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775690+0000", 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10029639899999999 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.775706+0000", 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6133e-05 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777345+0000", 2024-04-26T01:25:17.856 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016393899999999999 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:18.777421+0000", 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6021999999999998e-05 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=58, reqid=client.4589.0:337, at_version=0'0, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 58.257771890000001, 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.301426421, 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.857 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599661+0000", 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.858 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599662+0000", 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.299e-06 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599675+0000", 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2605e-05 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.599692+0000", 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7221999999999999e-05 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.859 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699796+0000", 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100104162 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699955+0000", 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000158319 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699983+0000", 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8453e-05 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.860 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901051+0000", 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201067725 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901088+0000", 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6636e-05 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 48/44 ECSubWrite(tid=60, reqid=osd.5.0:60, at_version=48'35, trim_to=0'0, roll_forward_to=48'33))", 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:17.861 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 58.157488473999997, 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20186222600000001, 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:17.862 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699945+0000", 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9200000000000002e-07 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699973+0000", 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8328999999999999e-05 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699980+0000", 2024-04-26T01:25:17.863 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0770000000000002e-06 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.699990+0000", 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.011e-05 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900620+0000", 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20062928899999999 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.864 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.900633+0000", 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2994e-05 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901770+0000", 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001137482 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:19.901807+0000", 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6653000000000003e-05 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.865 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=146, reqid=osd.14.0:146, at_version=67'60, trim_to=0'0, roll_forward_to=61'58))", 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.066224411, 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.200985516, 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.866 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791209+0000", 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3e-07 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791259+0000", 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9898999999999998e-05 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.867 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.791388+0000", 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012909999999999999 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.891494+0000", 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010556600000001 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.990490+0000", 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098996585999999998 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.868 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.990520+0000", 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0076000000000001e-05 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.992124+0000", 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016037709999999999 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:38.992194+0000", 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9888000000000006e-05 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.869 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 67/44 ECSubWrite(tid=113, reqid=osd.5.0:113, at_version=67'78, trim_to=0'0, roll_forward_to=67'76))", 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.717530248000003, 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.201983458, 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:25:17.870 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139903+0000", 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139905+0000", 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.86e-06 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139946+0000", 2024-04-26T01:25:17.871 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0948999999999997e-05 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139975+0000", 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9130000000000001e-05 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.139990+0000", 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5712000000000001e-05 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.238557+0000", 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.098566661999999999 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.872 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.238587+0000", 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9757e-05 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341833+0000", 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.103246136 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341886+0000", 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3251999999999998e-05 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.873 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 67/44 ECSubWrite(tid=114, reqid=osd.5.0:114, at_version=67'80, trim_to=0'0, roll_forward_to=67'76))", 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.715297749999998, 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.199785569, 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.874 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142135+0000", 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142137+0000", 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.843e-06 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142182+0000", 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4796000000000003e-05 2024-04-26T01:25:17.875 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142211+0000", 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9394999999999999e-05 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.142227+0000", 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5630000000000001e-05 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239528+0000", 2024-04-26T01:25:17.876 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.097300638999999994 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239543+0000", 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5616000000000001e-05 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341893+0000", 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102349381 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341921+0000", 2024-04-26T01:25:17.877 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8269e-05 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=150, reqid=osd.14.0:147, at_version=0'0, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.661596637000002, 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.146116522, 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.878 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195836+0000", 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.879 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195838+0000", 2024-04-26T01:25:17.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.03e-06 2024-04-26T01:25:17.881 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.881 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.881 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.881 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195859+0000", 2024-04-26T01:25:17.881 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0988999999999998e-05 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195891+0000", 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1962999999999999e-05 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.195911+0000", 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0046000000000001e-05 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.882 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239504+0000", 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.043592858999999998 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.239529+0000", 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5086999999999999e-05 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341923+0000", 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102394075 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.883 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341953+0000", 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9473000000000001e-05 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=67'64, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.659595054, 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14414429500000001, 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.884 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197838+0000", 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197840+0000", 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7269999999999999e-06 2024-04-26T01:25:17.885 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197895+0000", 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5092e-05 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197934+0000", 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8846e-05 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.886 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.197958+0000", 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3989e-05 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.340310+0000", 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.142352647 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.340336+0000", 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5992000000000001e-05 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.887 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341958+0000", 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016213429999999999 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341982+0000", 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4658999999999999e-05 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=152, reqid=osd.14.0:152, at_version=67'66, trim_to=0'0, roll_forward_to=67'62))", 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.657926433, 2024-04-26T01:25:17.888 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.142969076, 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199507+0000", 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.889 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199509+0000", 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1270000000000001e-06 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199563+0000", 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4565999999999999e-05 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199599+0000", 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6168999999999999e-05 2024-04-26T01:25:17.890 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.199623+0000", 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3275e-05 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341198+0000", 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.14157494500000001 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.341226+0000", 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8629999999999999e-05 2024-04-26T01:25:17.891 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.342404+0000", 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011773689999999999 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.342476+0000", 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1995000000000004e-05 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.892 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.3s3 67/63 ECSubWrite(tid=267, reqid=osd.13.0:265, at_version=0'0, trim_to=0'0, roll_forward_to=67'7))", 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.154863603000003, 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20128004799999999, 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.893 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702569+0000", 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702572+0000", 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.52e-06 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702594+0000", 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2166000000000001e-05 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.894 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.702623+0000", 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8960000000000001e-05 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.802711+0000", 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100088178 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.902926+0000", 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10021514500000001 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.895 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.902945+0000", 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8510000000000001e-05 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.903760+0000", 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00081536499999999997 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.903849+0000", 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9203999999999995e-05 2024-04-26T01:25:17.896 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=156, reqid=osd.14.0:152, at_version=0'0, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.056730283999997, 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.302725417, 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.897 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800703+0000", 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800704+0000", 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6869999999999999e-06 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.898 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800726+0000", 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1379e-05 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800752+0000", 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6472999999999999e-05 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.800766+0000", 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3441999999999999e-05 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.899 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.901012+0000", 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100246591 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.901028+0000", 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.611e-05 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103277+0000", 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20224862199999999 2024-04-26T01:25:17.900 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103428+0000", 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151113 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=157, reqid=osd.14.0:157, at_version=67'68, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.054193564000002, 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.30024277599999999, 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.901 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803239+0000", 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.902 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803247+0000", 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5569999999999998e-06 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803288+0000", 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0778999999999997e-05 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803309+0000", 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1342999999999999e-05 2024-04-26T01:25:17.903 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.803326+0000", 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7214000000000002e-05 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.001773+0000", 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19844687899999999 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.001789+0000", 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5449999999999999e-05 2024-04-26T01:25:17.904 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103444+0000", 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.101655572 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103482+0000", 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7982000000000003e-05 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.905 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 67/44 ECSubWrite(tid=158, reqid=osd.14.0:158, at_version=67'70, trim_to=0'0, roll_forward_to=67'66))", 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 38.051733714000001, 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.29805379300000001, 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.906 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805699+0000", 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805700+0000", 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.105e-06 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805744+0000", 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4044e-05 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.907 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805766+0000", 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1772000000000001e-05 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:39.805778+0000", 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.166e-05 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.102518+0000", 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.296740585 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.102534+0000", 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5571999999999999e-05 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103634+0000", 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011001979999999999 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:40.103753+0000", 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000118857 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.909 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 72/44 ECSubWrite(tid=197, reqid=osd.14.0:196, at_version=0'0, trim_to=0'0, roll_forward_to=72'93))", 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 33.076522033000003, 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20120590699999999, 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.910 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780911+0000", 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780912+0000", 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.373e-06 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.911 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780936+0000", 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3869000000000001e-05 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.780967+0000", 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0565999999999998e-05 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.881085+0000", 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100118531 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.912 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.881183+0000", 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7312999999999995e-05 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.881196+0000", 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3767999999999999e-05 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.982075+0000", 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10087816300000001 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.913 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:44.982117+0000", 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2324000000000003e-05 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 74/44 ECSubWrite(tid=200, reqid=client.4589.0:626, at_version=74'97, trim_to=0'0, roll_forward_to=72'95))", 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.926175069999999, 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202684053, 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.914 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931258+0000", 2024-04-26T01:25:17.915 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4000000000000002e-07 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931308+0000", 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9568999999999999e-05 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931342+0000", 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4266e-05 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.916 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.931368+0000", 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5409999999999999e-05 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.031641+0000", 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100273353 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.031657+0000", 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5713e-05 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.917 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133885+0000", 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.102228584 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133942+0000", 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6617999999999999e-05 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 74/44 ECSubWrite(tid=202, reqid=client.4589.0:627, at_version=74'98, trim_to=0'0, roll_forward_to=72'95))", 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:45.940804+0000", 2024-04-26T01:25:17.918 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.916628580000001, 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19317337600000001, 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940804+0000", 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940804+0000", 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.919 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940806+0000", 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6700000000000001e-06 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940842+0000", 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5775999999999998e-05 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940871+0000", 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9119e-05 2024-04-26T01:25:17.920 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.940887+0000", 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5625999999999999e-05 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.033111+0000", 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.092224414000000005 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.033125+0000", 2024-04-26T01:25:17.921 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4166e-05 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133949+0000", 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10082390200000001 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133978+0000", 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8702999999999999e-05 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.922 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 74/44 ECSubWrite(tid=203, reqid=client.4589.0:628, at_version=74'99, trim_to=0'0, roll_forward_to=72'95))", 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:45.943749+0000", 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.913684480000001, 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19124211899999999, 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943749+0000", 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.923 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943749+0000", 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943750+0000", 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-06 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943807+0000", 2024-04-26T01:25:17.924 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6759000000000003e-05 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943837+0000", 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9549000000000001e-05 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:45.943852+0000", 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5336999999999999e-05 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133244+0000", 2024-04-26T01:25:17.925 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.189391529 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.133275+0000", 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1900000000000003e-05 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.134910+0000", 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0016348529999999999 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.926 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:46.134991+0000", 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0371999999999996e-05 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 78/44 ECSubWrite(tid=228, reqid=osd.14.0:226, at_version=0'0, trim_to=0'0, roll_forward_to=78'117))", 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:24:50.739408+0000", 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 27.118024712, 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.202026329, 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.927 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739408+0000", 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739408+0000", 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739410+0000", 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8929999999999999e-06 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.928 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739432+0000", 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1413e-05 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.739463+0000", 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1891999999999999e-05 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.839568+0000", 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10010424800000001 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.929 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.939843+0000", 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100275244 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.939869+0000", 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6499000000000002e-05 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.941377+0000", 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015075310000000001 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.930 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:24:50.941435+0000", 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7608999999999997e-05 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s5 99/91 ECSubWrite(tid=192, reqid=client.4589.0:889, at_version=0'0, trim_to=0'0, roll_forward_to=99'136))", 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:11.941117+0000", 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.9163158129999998, 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.17376029700000001, 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.931 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.941117+0000", 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.941117+0000", 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.941120+0000", 2024-04-26T01:25:17.932 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4200000000000001e-06 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.941154+0000", 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4263e-05 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:11.941185+0000", 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1393000000000003e-05 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.933 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.041321+0000", 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.10013618000000001 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.113797+0000", 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.072475761 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.113820+0000", 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2629999999999998e-05 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.934 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.114763+0000", 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00094350800000000002 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.114877+0000", 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011414200000000001 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.935 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s2 100/44 ECSubWrite(tid=290, reqid=osd.14.0:289, at_version=0'0, trim_to=0'0, roll_forward_to=100'171))", 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-04-26T01:25:12.755511+0000", 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.101922461, 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.20119895900000001, 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.755511+0000", 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-04-26T01:25:17.936 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.755511+0000", 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.755512+0000", 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9740000000000001e-06 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.755534+0000", 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1892999999999999e-05 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-04-26T01:25:17.937 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.755565+0000", 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1025999999999997e-05 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.855665+0000", 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100099458 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.855840+0000", 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017530399999999999 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.938 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.855869+0000", 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8413000000000001e-05 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.956670+0000", 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.100801345 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-04-26T01:25:12.956709+0000", 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9545999999999997e-05 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.939 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.940 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-04-26T01:25:17.940 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-04-26T01:25:17.940 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-04-26T01:25:17.942 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:384: finishing write tid 6 to smithi18128752-15 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:384: oid 15 updating version 0 to 139 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:384: oid 15 updating version 139 to 164 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:384: oid 15 updating version 164 to 166 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:384: oid 15 version 166 is already newer than 165 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:384: oid 15 version 166 is already newer than 138 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 166 (ObjNum 139 snap 36 seq_num 139) dirty exists 2024-04-26T01:25:17.942 INFO:tasks.rados.rados.0.smithi181.stdout:384: left oid 15 (ObjNum 139 snap 36 seq_num 139) 2024-04-26T01:25:17.943 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-26T01:25:17.943+0000 7f7d08b67640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25062) UID: 0 2024-04-26T01:25:17.946 INFO:tasks.rados.rados.0.smithi181.stdout:386: finishing write tid 5 to smithi18128752-34 2024-04-26T01:25:18.035 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 13, 14, 15, 7, 12] out_osds: [2] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-04-26T01:25:18.036 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T01:25:18.036 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-04-26T01:25:18.036 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:25:18.042 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:18.046+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:18.068 INFO:tasks.rados.rados.0.smithi181.stdout:384: done (4 left) 2024-04-26T01:25:18.068 INFO:tasks.rados.rados.0.smithi181.stdout:392: done (3 left) 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:393: rollback oid 40 current snap is 38 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 40 to 32 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:394: rollback oid 50 current snap is 38 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 50 to 34 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:395: rmattr oid 48 current snap is 38 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:395: done (5 left) 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:396: copy_from oid 14 from oid 21 current snap is 38 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 19 current snap is 38 2024-04-26T01:25:18.069 INFO:tasks.rados.rados.0.smithi181.stdout:397: seq_num 142 ranges {0=442368,442368=720896,1163264=540672,1703936=688128} 2024-04-26T01:25:18.075 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:25:18.076 INFO:tasks.rados.rados.0.smithi181.stdout:397: writing smithi18128752-19 from 0 to 442368 tid 1 2024-04-26T01:25:18.086 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:18.086 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:18.086 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:25:18.086 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:18.087 INFO:tasks.rados.rados.0.smithi181.stdout:397: writing smithi18128752-19 from 442368 to 1163264 tid 2 2024-04-26T01:25:18.096 INFO:tasks.rados.rados.0.smithi181.stdout:397: writing smithi18128752-19 from 1163264 to 1703936 tid 3 2024-04-26T01:25:18.103 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-26T01:25:18.108 INFO:tasks.rados.rados.0.smithi181.stdout:397: writing smithi18128752-19 from 1703936 to 2392064 tid 4 2024-04-26T01:25:18.109 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 12 current snap is 38 2024-04-26T01:25:18.109 INFO:tasks.rados.rados.0.smithi181.stdout:398: seq_num 143 ranges {3293184=458752,3751936=475136,4227072=655360,4882432=671744,5554176=147456} 2024-04-26T01:25:18.143 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-04-26T01:25:18.148+0000 7fef29fa7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 25320) UID: 0 2024-04-26T01:25:18.151 INFO:tasks.rados.rados.0.smithi181.stdout:398: writing smithi18128752-12 from 3293184 to 3751936 tid 1 2024-04-26T01:25:18.154 INFO:tasks.rados.rados.0.smithi181.stdout:398: writing smithi18128752-12 from 3751936 to 4227072 tid 2 2024-04-26T01:25:18.159 INFO:tasks.rados.rados.0.smithi181.stdout:398: writing smithi18128752-12 from 4227072 to 4882432 tid 3 2024-04-26T01:25:18.164 INFO:tasks.rados.rados.0.smithi181.stdout:398: writing smithi18128752-12 from 4882432 to 5554176 tid 4 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:398: writing smithi18128752-12 from 5554176 to 5701632 tid 5 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:399: setattr oid 39 current snap is 38 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:386: finishing write tid 6 to smithi18128752-34 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:386: oid 34 updating version 0 to 140 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:386: oid 34 version 140 is already newer than 137 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:386: oid 34 updating version 140 to 141 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:386: oid 34 version 141 is already newer than 139 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:386: oid 34 version 141 is already newer than 138 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 141 (ObjNum 140 snap 36 seq_num 140) dirty exists 2024-04-26T01:25:18.166 INFO:tasks.rados.rados.0.smithi181.stdout:386: left oid 34 (ObjNum 140 snap 36 seq_num 140) 2024-04-26T01:25:18.168 INFO:tasks.rados.rados.0.smithi181.stdout:386: done (8 left) 2024-04-26T01:25:18.168 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 8 2024-04-26T01:25:18.244 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:18.246+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:18.273 INFO:tasks.rados.rados.0.smithi181.stdout:391: finishing rollback tid 1 to smithi18128752-45 2024-04-26T01:25:18.273 INFO:tasks.rados.rados.0.smithi181.stdout:391: finishing rollback tid 2 to smithi18128752-45 2024-04-26T01:25:18.273 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 175 (ObjNum 121 snap 30 seq_num 121) dirty exists 2024-04-26T01:25:18.273 INFO:tasks.rados.rados.0.smithi181.stdout:391: done (7 left) 2024-04-26T01:25:18.273 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 7 2024-04-26T01:25:18.278 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:18.278 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:18.278 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:25:18.278 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:25:18.278 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:18.293 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-26T01:25:18.345 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:18.347+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:18.366 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:18.367 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:25:18.374 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 39 v 45 (ObjNum 33 snap 0 seq_num 3031856096) dirty exists 2024-04-26T01:25:18.374 INFO:tasks.rados.rados.0.smithi181.stdout:399: done (6 left) 2024-04-26T01:25:18.375 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 6 2024-04-26T01:25:18.386 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-04-26T01:25:18.067599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":124879392,"num_objects":71,"num_object_clones":25,"num_object_copies":487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":4,"num_read":585,"num_read_kb":148482,"num_write":880,"num_write_kb":168804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":62640672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1464,"ondisk_log_size":1464,"up":144,"acting":144,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":704344,"kb_used_data":232468,"kb_used_omap":23,"kb_used_meta":471656,"kb_avail":1414873256,"statfs":{"total":1449551462400,"available":1448830214144,"internally_reserved":0,"allocated":238047232,"data_stored":292549492,"data_compressed":3641903,"data_compressed_allocated":4993024,"data_compressed_original":10858496,"omap_allocated":23926,"internal_metadata":482976394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,3,4,8,5,10],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6914048,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":4801,"num_write":14,"num_write_kb":6752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3391488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039414"},"pg_stats":[{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"105'141","reported_seq":583,"reported_epoch":105,"state":"active+recovering+degraded","last_fresh":"2024-04-26T01:25:17.539125+0000","last_change":"2024-04-26T01:25:16.497881+0000","last_active":"2024-04-26T01:25:17.539125+0000","last_peered":"2024-04-26T01:25:17.539125+0000","last_clean":"2024-04-26T01:25:01.261092+0000","last_became_active":"2024-04-26T01:25:15.750526+0000","last_became_peered":"2024-04-26T01:25:15.750526+0000","last_unstale":"2024-04-26T01:25:17.539125+0000","last_undegraded":"2024-04-26T01:25:15.446828+0000","last_fullsized":"2024-04-26T01:25:17.539125+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":26,"num_read_kb":13291,"num_write":58,"num_write_kb":13592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":7766016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,12,1,14,15,6],"acting":[5,0,12,1,14,15,6],"avail_no_missing":["5(0)","0(1)","1(2)","6(6)","9(3)","14(4)","15(5)"],"object_location_counts":[{"shards":"0(1),1(3),5(0),6(6),12(2),14(4),15(5)","objects":4},{"shards":"0(1),5(0),6(6),14(4),15(5)","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"100'72","reported_seq":443,"reported_epoch":101,"state":"unknown","last_fresh":"2024-04-26T01:25:13.266491+0000","last_change":"2024-04-26T01:25:14.252120+0000","last_active":"2024-04-26T01:25:14.252120+0000","last_peered":"2024-04-26T01:25:13.266491+0000","last_clean":"2024-04-26T01:25:13.266491+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:25:13.266491+0000","last_undegraded":"2024-04-26T01:25:13.266491+0000","last_fullsized":"2024-04-26T01:25:13.266491+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8019968,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":7688,"num_write":18,"num_write_kb":8072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":157,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-26T01:25:12.014120+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:25:12.014120+0000","last_peered":"2024-04-26T01:25:12.014120+0000","last_clean":"2024-04-26T01:25:12.014120+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:12.014120+0000","last_undegraded":"2024-04-26T01:25:12.014120+0000","last_fullsized":"2024-04-26T01:25:12.014120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"72'37","reported_seq":428,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-26T01:25:12.019312+0000","last_change":"2024-04-26T01:25:12.120275+0000","last_active":"2024-04-26T01:25:12.019312+0000","last_peered":"2024-04-26T01:25:12.019312+0000","last_clean":"2024-04-26T01:25:12.019312+0000","last_became_active":"2024-04-26T01:25:03.106948+0000","last_became_peered":"2024-04-26T01:25:03.106948+0000","last_unstale":"2024-04-26T01:25:12.019312+0000","last_undegraded":"2024-04-26T01:25:12.019312+0000","last_fullsized":"2024-04-26T01:25:12.019312+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101409713,"stat_sum":{"num_bytes":114688,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2645,"num_write":18,"num_write_kb":3392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":160,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-04-26T01:25:14.150137+0000","last_change":"2024-04-26T01:23:52.015661+0000","last_active":"2024-04-26T01:25:14.150137+0000","last_peered":"2024-04-26T01:25:14.150137+0000","last_clean":"2024-04-26T01:25:14.150137+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:14.150137+0000","last_undegraded":"2024-04-26T01:25:14.150137+0000","last_fullsized":"2024-04-26T01:25:14.150137+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:16.916052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"100'109","reported_seq":467,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.044767+0000","last_change":"2024-04-26T01:25:12.226659+0000","last_active":"2024-04-26T01:25:13.044767+0000","last_peered":"2024-04-26T01:25:13.044767+0000","last_clean":"2024-04-26T01:25:13.044767+0000","last_became_active":"2024-04-26T01:24:01.091412+0000","last_became_peered":"2024-04-26T01:24:01.091412+0000","last_unstale":"2024-04-26T01:25:13.044767+0000","last_undegraded":"2024-04-26T01:25:13.044767+0000","last_fullsized":"2024-04-26T01:25:13.044767+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'103","last_scrub_stamp":"2024-04-26T01:25:08.533710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:08.533710+0000","objects_scrubbed":4,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:22.936714+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0.113530442,"stat_sum":{"num_bytes":11829248,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":5632,"num_write":64,"num_write_kb":13216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":166,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.137547+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:25:17.137547+0000","last_peered":"2024-04-26T01:25:17.137547+0000","last_clean":"2024-04-26T01:25:17.137547+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:17.137547+0000","last_undegraded":"2024-04-26T01:25:17.137547+0000","last_fullsized":"2024-04-26T01:25:17.137547+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"100'72","reported_seq":443,"reported_epoch":101,"state":"peering","last_fresh":"2024-04-26T01:25:13.266491+0000","last_change":"2024-04-26T01:25:14.140238+0000","last_active":"2024-04-26T01:25:14.140238+0000","last_peered":"2024-04-26T01:25:13.266491+0000","last_clean":"2024-04-26T01:25:13.266491+0000","last_became_active":"2024-04-26T01:24:16.445774+0000","last_became_peered":"2024-04-26T01:24:16.445774+0000","last_unstale":"2024-04-26T01:25:13.266491+0000","last_undegraded":"2024-04-26T01:25:13.266491+0000","last_fullsized":"2024-04-26T01:25:13.266491+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20506823599999999,"stat_sum":{"num_bytes":8019968,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":7687,"num_write":18,"num_write_kb":8072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":745472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,4,9,13,14],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":166,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.044613+0000","last_change":"2024-04-26T01:25:10.080365+0000","last_active":"2024-04-26T01:25:13.044613+0000","last_peered":"2024-04-26T01:25:13.044613+0000","last_clean":"2024-04-26T01:25:13.044613+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:13.044613+0000","last_undegraded":"2024-04-26T01:25:13.044613+0000","last_fullsized":"2024-04-26T01:25:13.044613+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:10.080096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:10.080096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.198292+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":353,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-04-26T01:25:12.014716+0000","last_change":"2024-04-26T01:25:12.015627+0000","last_active":"2024-04-26T01:25:12.014716+0000","last_peered":"2024-04-26T01:25:12.014716+0000","last_clean":"2024-04-26T01:25:12.014716+0000","last_became_active":"2024-04-26T01:24:36.199837+0000","last_became_peered":"2024-04-26T01:24:36.199837+0000","last_unstale":"2024-04-26T01:25:12.014716+0000","last_undegraded":"2024-04-26T01:25:12.014716+0000","last_fullsized":"2024-04-26T01:25:12.014716+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014197909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"3.3","version":"100'15","reported_seq":387,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.019053+0000","last_change":"2024-04-26T01:25:12.425289+0000","last_active":"2024-04-26T01:25:13.019053+0000","last_peered":"2024-04-26T01:25:13.019053+0000","last_clean":"2024-04-26T01:25:13.019053+0000","last_became_active":"2024-04-26T01:24:36.104415+0000","last_became_peered":"2024-04-26T01:24:36.104415+0000","last_unstale":"2024-04-26T01:25:13.019053+0000","last_undegraded":"2024-04-26T01:25:13.019053+0000","last_fullsized":"2024-04-26T01:25:13.019053+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27352768399999999,"stat_sum":{"num_bytes":2981888,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":4532,"num_write":24,"num_write_kb":3520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,6,5,3],"acting":[13,7,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.294950+0000","last_change":"2024-04-26T01:25:02.920443+0000","last_active":"2024-04-26T01:25:17.294950+0000","last_peered":"2024-04-26T01:25:17.294950+0000","last_clean":"2024-04-26T01:25:17.294950+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:17.294950+0000","last_undegraded":"2024-04-26T01:25:17.294950+0000","last_fullsized":"2024-04-26T01:25:17.294950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.351681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"100'171","reported_seq":537,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.171985+0000","last_change":"2024-04-26T01:25:12.756196+0000","last_active":"2024-04-26T01:25:13.171985+0000","last_peered":"2024-04-26T01:25:13.171985+0000","last_clean":"2024-04-26T01:25:13.171985+0000","last_became_active":"2024-04-26T01:24:16.549243+0000","last_became_peered":"2024-04-26T01:24:16.549243+0000","last_unstale":"2024-04-26T01:25:13.171985+0000","last_undegraded":"2024-04-26T01:25:13.171985+0000","last_fullsized":"2024-04-26T01:25:13.171985+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:26.911169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.73509105299999999,"stat_sum":{"num_bytes":7880704,"num_objects":12,"num_object_clones":5,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":54,"num_read_kb":23878,"num_write":105,"num_write_kb":14208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":159,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.018853+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:25:13.018853+0000","last_peered":"2024-04-26T01:25:13.018853+0000","last_clean":"2024-04-26T01:25:13.018853+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:13.018853+0000","last_undegraded":"2024-04-26T01:25:13.018853+0000","last_fullsized":"2024-04-26T01:25:13.018853+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.152303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"101'65","reported_seq":451,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.355866+0000","last_change":"2024-04-26T01:25:12.222031+0000","last_active":"2024-04-26T01:25:13.355866+0000","last_peered":"2024-04-26T01:25:13.355866+0000","last_clean":"2024-04-26T01:25:13.355866+0000","last_became_active":"2024-04-26T01:25:03.216745+0000","last_became_peered":"2024-04-26T01:25:03.216745+0000","last_unstale":"2024-04-26T01:25:13.355866+0000","last_undegraded":"2024-04-26T01:25:13.355866+0000","last_fullsized":"2024-04-26T01:25:13.355866+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20937640599999999,"stat_sum":{"num_bytes":15204352,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":11804,"num_write":38,"num_write_kb":21408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":5357568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,12,13],"acting":[4,11,9,5,10,12,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":174,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.294820+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:17.294820+0000","last_peered":"2024-04-26T01:25:17.294820+0000","last_clean":"2024-04-26T01:25:17.294820+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:17.294820+0000","last_undegraded":"2024-04-26T01:25:17.294820+0000","last_fullsized":"2024-04-26T01:25:17.294820+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'68","reported_seq":435,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.119138+0000","last_change":"2024-04-26T01:25:12.017961+0000","last_active":"2024-04-26T01:25:13.119138+0000","last_peered":"2024-04-26T01:25:13.119138+0000","last_clean":"2024-04-26T01:25:13.119138+0000","last_became_active":"2024-04-26T01:24:16.348216+0000","last_became_peered":"2024-04-26T01:24:16.348216+0000","last_unstale":"2024-04-26T01:25:13.119138+0000","last_undegraded":"2024-04-26T01:25:13.119138+0000","last_fullsized":"2024-04-26T01:25:13.119138+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'68","last_scrub_stamp":"2024-04-26T01:25:09.880728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:09.880728+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:26.536874+0000","scrub_duration":815,"objects_trimmed":0,"snaptrim_duration":0.00072304499999999996,"stat_sum":{"num_bytes":12419072,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":1210,"num_write":31,"num_write_kb":12848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":167,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.364027+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:25:17.364027+0000","last_peered":"2024-04-26T01:25:17.364027+0000","last_clean":"2024-04-26T01:25:17.364027+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:17.364027+0000","last_undegraded":"2024-04-26T01:25:17.364027+0000","last_fullsized":"2024-04-26T01:25:17.364027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":379,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.118912+0000","last_change":"2024-04-26T01:24:56.353548+0000","last_active":"2024-04-26T01:25:13.118912+0000","last_peered":"2024-04-26T01:25:13.118912+0000","last_clean":"2024-04-26T01:25:13.118912+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:25:13.118912+0000","last_undegraded":"2024-04-26T01:25:13.118912+0000","last_fullsized":"2024-04-26T01:25:13.118912+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:10.087788+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"104'143","reported_seq":586,"reported_epoch":105,"state":"active+recovering+degraded","last_fresh":"2024-04-26T01:25:17.534733+0000","last_change":"2024-04-26T01:25:17.640744+0000","last_active":"2024-04-26T01:25:17.534733+0000","last_peered":"2024-04-26T01:25:17.534733+0000","last_clean":"2024-04-26T01:25:01.261092+0000","last_became_active":"2024-04-26T01:25:16.396635+0000","last_became_peered":"2024-04-26T01:25:16.396635+0000","last_unstale":"2024-04-26T01:25:17.534733+0000","last_undegraded":"2024-04-26T01:25:16.057678+0000","last_fullsized":"2024-04-26T01:25:17.534733+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31276866399999997,"stat_sum":{"num_bytes":11771904,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":13291,"num_write":58,"num_write_kb":14232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":7766016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,12,1,14,15,6],"acting":[5,0,12,1,14,15,6],"avail_no_missing":["5(0)","0(1)","1(2)","6(6)","9(3)","14(4)","15(5)"],"object_location_counts":[{"shards":"0(1),1(3),5(0),6(6),12(2),14(4),15(5)","objects":5},{"shards":"0(1),5(0),6(6),14(4),15(5)","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"91'65","reported_seq":401,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.172010+0000","last_change":"2024-04-26T01:25:12.122662+0000","last_active":"2024-04-26T01:25:13.172010+0000","last_peered":"2024-04-26T01:25:13.172010+0000","last_clean":"2024-04-26T01:25:13.172010+0000","last_became_active":"2024-04-26T01:24:01.291346+0000","last_became_peered":"2024-04-26T01:24:01.291346+0000","last_unstale":"2024-04-26T01:25:13.172010+0000","last_undegraded":"2024-04-26T01:25:13.172010+0000","last_fullsized":"2024-04-26T01:25:13.172010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:24.270106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001504383,"stat_sum":{"num_bytes":5718016,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":1015,"num_write":24,"num_write_kb":7664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"3.a","version":"104'43","reported_seq":415,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.194825+0000","last_change":"2024-04-26T01:25:14.247759+0000","last_active":"2024-04-26T01:25:17.194825+0000","last_peered":"2024-04-26T01:25:17.194825+0000","last_clean":"2024-04-26T01:25:17.194825+0000","last_became_active":"2024-04-26T01:24:36.102041+0000","last_became_peered":"2024-04-26T01:24:36.102041+0000","last_unstale":"2024-04-26T01:25:17.194825+0000","last_undegraded":"2024-04-26T01:25:17.194825+0000","last_fullsized":"2024-04-26T01:25:17.194825+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'39","last_scrub_stamp":"2024-04-26T01:25:06.819166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:06.819166+0000","objects_scrubbed":1,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:07.075209+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0.001473491,"stat_sum":{"num_bytes":3293184,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":9684,"num_write":35,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.b","version":"72'28","reported_seq":421,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.044591+0000","last_change":"2024-04-26T01:25:12.015359+0000","last_active":"2024-04-26T01:25:13.044591+0000","last_peered":"2024-04-26T01:25:13.044591+0000","last_clean":"2024-04-26T01:25:13.044591+0000","last_became_active":"2024-04-26T01:25:03.018610+0000","last_became_peered":"2024-04-26T01:25:03.018610+0000","last_unstale":"2024-04-26T01:25:13.044591+0000","last_undegraded":"2024-04-26T01:25:13.044591+0000","last_fullsized":"2024-04-26T01:25:13.044591+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:15.345184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001840796,"stat_sum":{"num_bytes":1458176,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":4936,"num_write":30,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3407872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"3.c","version":"100'106","reported_seq":491,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.177424+0000","last_change":"2024-04-26T01:25:12.873480+0000","last_active":"2024-04-26T01:25:13.177424+0000","last_peered":"2024-04-26T01:25:13.177424+0000","last_clean":"2024-04-26T01:25:13.177424+0000","last_became_active":"2024-04-26T01:25:03.109927+0000","last_became_peered":"2024-04-26T01:25:03.109927+0000","last_unstale":"2024-04-26T01:25:13.177424+0000","last_undegraded":"2024-04-26T01:25:13.177424+0000","last_fullsized":"2024-04-26T01:25:13.177424+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71784870199999995,"stat_sum":{"num_bytes":11583488,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":3853,"num_write":82,"num_write_kb":15600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8650752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"3.d","version":"103'83","reported_seq":293,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.165186+0000","last_change":"2024-04-26T01:25:14.054116+0000","last_active":"2024-04-26T01:25:17.165186+0000","last_peered":"2024-04-26T01:25:17.165186+0000","last_clean":"2024-04-26T01:25:17.165186+0000","last_became_active":"2024-04-26T01:24:36.105677+0000","last_became_peered":"2024-04-26T01:24:36.105677+0000","last_unstale":"2024-04-26T01:25:17.165186+0000","last_undegraded":"2024-04-26T01:25:17.165186+0000","last_fullsized":"2024-04-26T01:25:17.165186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067180240000000004,"stat_sum":{"num_bytes":5554176,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":44,"num_read_kb":19501,"num_write":33,"num_write_kb":13936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.e","version":"100'79","reported_seq":395,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.019053+0000","last_change":"2024-04-26T01:25:12.258556+0000","last_active":"2024-04-26T01:25:13.019053+0000","last_peered":"2024-04-26T01:25:13.019053+0000","last_clean":"2024-04-26T01:25:13.019053+0000","last_became_active":"2024-04-26T01:25:03.422052+0000","last_became_peered":"2024-04-26T01:25:03.422052+0000","last_unstale":"2024-04-26T01:25:13.019053+0000","last_undegraded":"2024-04-26T01:25:13.019053+0000","last_fullsized":"2024-04-26T01:25:13.019053+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:29.817028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.24100895899999999,"stat_sum":{"num_bytes":6569984,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":43,"num_read_kb":13824,"num_write":35,"num_write_kb":9008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,12],"acting":[13,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"3.f","version":"96'10","reported_seq":363,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-04-26T01:25:13.177514+0000","last_change":"2024-04-26T01:25:12.220869+0000","last_active":"2024-04-26T01:25:13.177514+0000","last_peered":"2024-04-26T01:25:13.177514+0000","last_clean":"2024-04-26T01:25:13.177514+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:25:13.177514+0000","last_undegraded":"2024-04-26T01:25:13.177514+0000","last_fullsized":"2024-04-26T01:25:13.177514+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101664559,"stat_sum":{"num_bytes":753664,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3893,"num_write":6,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T01:25:14.016590+0000","last_change":"2024-04-26T01:25:14.016590+0000","last_active":"2024-04-26T01:25:14.016590+0000","last_peered":"2024-04-26T01:25:14.016590+0000","last_clean":"2024-04-26T01:25:14.016590+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T01:25:14.016590+0000","last_undegraded":"2024-04-26T01:25:14.016590+0000","last_fullsized":"2024-04-26T01:25:14.016590+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:25:14.016590+0000","last_clean_scrub_stamp":"2024-04-26T01:25:14.016590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":124289024,"num_objects":69,"num_object_clones":25,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":4,"num_read":449,"num_read_kb":148364,"num_write":677,"num_write_kb":165344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":62111744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":226476032,"data_stored":287940608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1307,"ondisk_log_size":1307,"up":126,"acting":126,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45692,"kb_used_data":14516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326148,"statfs":{"total":96636764160,"available":96589975552,"internally_reserved":0,"allocated":14864384,"data_stored":17204631,"data_compressed":235535,"data_compressed_allocated":323584,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280531,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46496,"kb_used_data":15316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325344,"statfs":{"total":96636764160,"available":96589152256,"internally_reserved":0,"allocated":15683584,"data_stored":18376087,"data_compressed":235535,"data_compressed_allocated":323584,"data_compressed_original":647168,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49540,"kb_used_data":18360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322300,"statfs":{"total":96636764160,"available":96586035200,"internally_reserved":0,"allocated":18800640,"data_stored":25206021,"data_compressed":232509,"data_compressed_allocated":319488,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313235,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52300,"kb_used_data":21120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319540,"statfs":{"total":96636764160,"available":96583208960,"internally_reserved":0,"allocated":21626880,"data_stored":24788229,"data_compressed":232509,"data_compressed_allocated":319488,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313236,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46168,"kb_used_data":14988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325672,"statfs":{"total":96636764160,"available":96589488128,"internally_reserved":0,"allocated":15347712,"data_stored":19270917,"data_compressed":232509,"data_compressed_allocated":319488,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313237,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313235,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43700,"kb_used_data":12516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328140,"statfs":{"total":96636764160,"available":96592015360,"internally_reserved":0,"allocated":12816384,"data_stored":18046213,"data_compressed":232509,"data_compressed_allocated":319488,"data_compressed_original":638976,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39732,"kb_used_data":8544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":8749056,"data_stored":10396212,"data_compressed":278646,"data_compressed_allocated":376832,"data_compressed_original":1196032,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313235,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46644,"kb_used_data":15464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325196,"statfs":{"total":96636764160,"available":96589000704,"internally_reserved":0,"allocated":15835136,"data_stored":17063173,"data_compressed":232509,"data_compressed_allocated":319488,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,1,1,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55648,"kb_used_data":20308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316192,"statfs":{"total":96636764160,"available":96579780608,"internally_reserved":0,"allocated":20795392,"data_stored":27076741,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52216,"kb_used_data":21040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319624,"statfs":{"total":96636764160,"available":96583294976,"internally_reserved":0,"allocated":21544960,"data_stored":30562437,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313237,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":5016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":5136384,"data_stored":5666949,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313236,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48876,"kb_used_data":17692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322964,"statfs":{"total":96636764160,"available":96586715136,"internally_reserved":0,"allocated":18116608,"data_stored":20620695,"data_compressed":235535,"data_compressed_allocated":323584,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313236,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46336,"kb_used_data":15148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325504,"statfs":{"total":96636764160,"available":96589316096,"internally_reserved":0,"allocated":15511552,"data_stored":17684613,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49304,"kb_used_data":18120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322536,"statfs":{"total":96636764160,"available":96586276864,"internally_reserved":0,"allocated":18554880,"data_stored":23598487,"data_compressed":235535,"data_compressed_allocated":323584,"data_compressed_original":647168,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,0,0,0,0,5,1,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313236,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49660,"kb_used_data":14320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322180,"statfs":{"total":96636764160,"available":96585912320,"internally_reserved":0,"allocated":14663680,"data_stored":16988087,"data_compressed":268152,"data_compressed_allocated":364544,"data_compressed_original":1159168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17641472,"data_stored":24338432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20992000,"data_stored":24444928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7491584,"data_stored":9781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":15237120,"data_stored":17305600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19574784,"data_stored":26677248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20324352,"data_stored":30683136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3915776,"data_stored":4747264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":17498112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16945152,"data_stored":19742720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":14290944,"data_stored":16764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17383424,"data_stored":22720512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":14479360,"data_stored":17113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":13520896,"data_stored":15671296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14188544,"data_stored":18403328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":11657216,"data_stored":17178624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:25:18.389 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-04-26T01:25:18.389 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-04-26T01:25:18.418 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:18.418 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:25:18.418 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:116 3.6s0 3:69f31b93:::smithi18128752-39:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 73.035740028000006, 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.37357116200000001, 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 116 2024-04-26T01:25:18.419 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386855+0000", 2024-04-26T01:25:18.420 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386857+0000", 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.703e-06 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386866+0000", 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5670000000000001e-06 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386887+0000", 2024-04-26T01:25:18.421 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.1316000000000001e-05 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386904+0000", 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6767000000000001e-05 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.556964+0000", 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17005975700000001 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.422 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.557176+0000", 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00021186600000000001 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558110+0000", 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00093395700000000002 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.559561+0000", 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0014508430000000001 2024-04-26T01:25:18.423 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.759894+0000", 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033284600000001 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.760426+0000", 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00053253999999999997 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.424 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:117 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 73.035665374999994, 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77605625899999997, 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 117 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.425 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386930+0000", 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386931+0000", 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3599999999999999e-06 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.426 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386939+0000", 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8369999999999997e-06 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386957+0000", 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8683000000000001e-05 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.386974+0000", 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6390000000000001e-05 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.427 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558747+0000", 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.17177335099999999 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.558933+0000", 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00018625099999999999 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761103+0000", 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202169231 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.428 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.761231+0000", 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000128734 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.162986+0000", 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.401754422 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:156 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~475136] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:25:18.429 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 71.924959631999997, 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33900037700000002, 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 156 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.430 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497635+0000", 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.497638+0000", 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2110000000000001e-06 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.431 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498467+0000", 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082923399999999998 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498486+0000", 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9341e-05 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498496+0000", 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8179999999999999e-06 2024-04-26T01:25:18.432 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498610+0000", 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000114259 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.498770+0000", 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016017599999999999 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726262+0000", 2024-04-26T01:25:18.433 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22749161900000001 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.726429+0000", 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000166825 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.728061+0000", 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016325490000000001 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.434 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735773+0000", 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0077114189999999997 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836407+0000", 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10063380399999999 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836636+0000", 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00022912200000000001 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.435 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:157 3.6s0 3:60f0095a:::smithi18128752-3:head [append 0~376832] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 71.918739165999995, 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.43381848000000001, 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 157 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.436 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503856+0000", 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.503857+0000", 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2920000000000001e-06 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.437 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504200+0000", 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00034286599999999999 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504218+0000", 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7961e-05 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504227+0000", 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.6240000000000008e-06 2024-04-26T01:25:18.438 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504381+0000", 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015489900000000001 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504555+0000", 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017342900000000001 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.729729+0000", 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.225174034 2024-04-26T01:25:18.439 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.730038+0000", 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00030876200000000002 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.732244+0000", 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0022065190000000001 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.735776+0000", 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0035319330000000001 2024-04-26T01:25:18.440 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937311+0000", 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.201534612 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.937674+0000", 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00036354900000000001 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:158 3.6s0 3:60f0095a:::smithi18128752-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.441 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 71.918348093999995, 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.630196439, 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 158 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:18.442 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504247+0000", 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.0500000000000004e-07 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504255+0000", 2024-04-26T01:25:18.443 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.0989999999999999e-06 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504262+0000", 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5660000000000003e-06 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504266+0000", 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0980000000000004e-06 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504650+0000", 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00038378400000000002 2024-04-26T01:25:18.444 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504796+0000", 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014592199999999999 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733685+0000", 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.22888902699999999 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.733975+0000", 2024-04-26T01:25:18.445 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029027399999999998 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.734990+0000", 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010148500000000001 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.836708+0000", 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10171771 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:18.446 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134326+0000", 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29761846199999997 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134443+0000", 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000117142 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:159 3.6s0 3:60f0095a:::smithi18128752-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 71.918322469000003, 2024-04-26T01:25:18.447 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.73307043299999997, 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 159 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.448 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504272+0000", 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504273+0000", 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3500000000000002e-07 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504274+0000", 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.562e-06 2024-04-26T01:25:18.449 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504281+0000", 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3790000000000003e-06 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504286+0000", 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1989999999999997e-06 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504861+0000", 2024-04-26T01:25:18.450 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00057502400000000004 2024-04-26T01:25:18.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.451 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.451 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for degraded object", 2024-04-26T01:25:18.451 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.504981+0000", 2024-04-26T01:25:18.451 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012016899999999999 2024-04-26T01:25:18.451 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.451 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837238+0000", 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.33225718900000001 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.837352+0000", 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000113583 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.452 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134907+0000", 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.29755462599999999 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.135101+0000", 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019402500000000001 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.453 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.237343+0000", 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102242342 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/34 ECSubWriteReply(tid=11, last_complete=35'7, committed=1, applied=1))", 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 71.588448403000001, 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30069599499999999, 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.454 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.455 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834147+0000", 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.0800000000000003e-07 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834153+0000", 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.1699999999999996e-06 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.834168+0000", 2024-04-26T01:25:18.456 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5017999999999999e-05 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.934244+0000", 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100075936 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134251+0000", 2024-04-26T01:25:18.457 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200007516 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.134843+0000", 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00059144700000000004 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:171 3.6s0 3:69f31b93:::smithi18128752-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.458 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 71.085899531999999, 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36489432300000002, 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 171 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.459 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336695+0000", 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336697+0000", 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3829999999999999e-06 2024-04-26T01:25:18.460 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336701+0000", 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.053e-06 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336710+0000", 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.8349999999999993e-06 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.461 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.336715+0000", 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5219999999999998e-06 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337072+0000", 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00035695799999999999 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.462 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.337245+0000", 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017283899999999999 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.338075+0000", 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00082985399999999999 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.339323+0000", 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012477390000000001 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.463 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701394+0000", 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.36207125499999998 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.701590+0000", 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000195885 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.464 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 51/44 ECSubWrite(tid=71, reqid=client.4589.0:385, at_version=0'0, trim_to=0'0, roll_forward_to=51'44))", 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 54.696186081, 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30170434200000001, 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.465 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726409+0000", 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726411+0000", 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.88e-06 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726432+0000", 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.137e-05 2024-04-26T01:25:18.466 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.726464+0000", 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1664e-05 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826555+0000", 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100091667 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826667+0000", 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00011122500000000001 2024-04-26T01:25:18.467 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:23.826683+0000", 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6149000000000001e-05 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028057+0000", 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20137464699999999 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:24.028113+0000", 2024-04-26T01:25:18.468 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5739999999999998e-05 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:431 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 50.621319702999998, 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40905669700000002, 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.469 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 431 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801275+0000", 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.470 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801276+0000", 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.1399999999999995e-07 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801283+0000", 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4840000000000001e-06 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801299+0000", 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6707000000000001e-05 2024-04-26T01:25:18.471 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801309+0000", 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0112000000000001e-05 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801486+0000", 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017675199999999999 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801494+0000", 2024-04-26T01:25:18.472 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6490000000000008e-06 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906250+0000", 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10475599300000001 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.906413+0000", 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016362200000000001 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.210332+0000", 2024-04-26T01:25:18.473 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303918464 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:432 3.6s0 3:69f31b93:::smithi18128752-39:head [read 0~901120,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56)", 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 50.621211774999999, 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.51044518999999999, 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:18.474 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 432 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801383+0000", 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.475 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801384+0000", 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5000000000000001e-07 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801390+0000", 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2750000000000001e-06 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.476 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.801402+0000", 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1747e-05 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:27.901507+0000", 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100104586 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.007942+0000", 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10643573100000001 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.477 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.008127+0000", 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000185007 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.311828+0000", 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.303700894 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.478 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpReadReply(3.6s0 56/34 ECSubReadReply(tid=40, attrs_read=0))", 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 50.413140661, 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30115782000000002, 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.479 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009454+0000", 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.009456+0000", 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7799999999999999e-06 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.480 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010014+0000", 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00055842399999999996 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010048+0000", 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3407e-05 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.010060+0000", 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.254e-05 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.481 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310534+0000", 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30047343599999998 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.310612+0000", 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8232999999999994e-05 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.482 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 71/44 ECSubWrite(tid=122, reqid=osd.5.0:122, at_version=71'86, trim_to=0'0, roll_forward_to=70'84))", 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 35.401036017999999, 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30196446599999999, 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:18.483 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021559+0000", 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.29e-07 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.484 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021579+0000", 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.0132e-05 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.021590+0000", 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0645000000000001e-05 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.485 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.121747+0000", 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100156917 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.322084+0000", 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20033731900000001 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.322100+0000", 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5780000000000001e-05 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.486 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.323445+0000", 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0013450269999999999 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.323523+0000", 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8417000000000005e-05 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:603 3.6s0 3:69f31b93:::smithi18128752-39:head [stat,delete] snapc 12=[12] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:25:18.487 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 35.064725842000001, 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40426880399999998, 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 603 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:25:18.488 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357869+0000", 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357872+0000", 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.207e-06 2024-04-26T01:25:18.489 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357887+0000", 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4443e-05 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357921+0000", 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3954000000000001e-05 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.490 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.357937+0000", 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5823000000000001e-05 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.358088+0000", 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00015174899999999999 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.358258+0000", 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169999 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.359596+0000", 2024-04-26T01:25:18.491 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001337378 2024-04-26T01:25:18.492 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.492 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.493 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.493 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.361439+0000", 2024-04-26T01:25:18.493 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0018429900000000001 2024-04-26T01:25:18.493 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.493 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.493 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:18.493 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.761928+0000", 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.400489284 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.762138+0000", 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000209977 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.7s3 71/30 ECSubWrite(tid=257, reqid=osd.4.0:257, at_version=71'80, trim_to=0'0, roll_forward_to=71'76))", 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:25:18.494 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 34.663196968000001, 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30249526599999998, 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759398+0000", 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.495 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759399+0000", 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2999999999999999e-07 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759444+0000", 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5158999999999999e-05 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759467+0000", 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2892999999999999e-05 2024-04-26T01:25:18.496 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.759485+0000", 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8369999999999999e-05 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.960407+0000", 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200921553 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:43.960423+0000", 2024-04-26T01:25:18.497 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6169999999999999e-05 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.061786+0000", 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101363014 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:44.061893+0000", 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010717699999999999 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.498 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/44 ECSubWrite(tid=144, reqid=client.4589.0:762, at_version=84'109, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.233496+0000", 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.189098427000001, 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30456070200000002, 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233496+0000", 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.499 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233496+0000", 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233498+0000", 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9800000000000001e-06 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233738+0000", 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023948599999999999 2024-04-26T01:25:18.500 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233776+0000", 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7993e-05 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.233799+0000", 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3309999999999999e-05 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.333994+0000", 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10019452199999999 2024-04-26T01:25:18.501 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.334009+0000", 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5624000000000002e-05 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.537978+0000", 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20396856099999999 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538057+0000", 2024-04-26T01:25:18.502 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.9226000000000003e-05 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/44 ECSubWrite(tid=145, reqid=client.4589.0:763, at_version=84'110, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.235255+0000", 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.187339984000001, 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30286903199999998, 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.503 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235255+0000", 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235255+0000", 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235257+0000", 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6330000000000001e-06 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.504 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235493+0000", 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000236333 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235523+0000", 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9810000000000001e-05 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.235538+0000", 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.485e-05 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.505 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.436361+0000", 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20082362000000001 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.436376+0000", 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5277e-05 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538071+0000", 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.101694746 2024-04-26T01:25:18.506 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538124+0000", 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2763000000000003e-05 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.8s1 84/44 ECSubWrite(tid=146, reqid=client.4589.0:764, at_version=84'111, trim_to=0'0, roll_forward_to=84'104))", 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.236721+0000", 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.18587359, 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30146896499999998, 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.507 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236721+0000", 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236721+0000", 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236723+0000", 2024-04-26T01:25:18.508 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8729999999999999e-06 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237026+0000", 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000302849 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237061+0000", 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.5006999999999999e-05 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237076+0000", 2024-04-26T01:25:18.509 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5393e-05 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.437332+0000", 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20025568299999999 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.437348+0000", 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.5699999999999999e-05 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.510 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538129+0000", 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100781433 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.538190+0000", 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.1026999999999997e-05 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.cs2 98/90 ECSubWrite(tid=148, reqid=client.4589.0:871, at_version=98'94, trim_to=0'0, roll_forward_to=94'90))", 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:10.784576+0000", 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.6380184260000004, 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30392043499999999, 2024-04-26T01:25:18.511 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "reached pg", 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.784576+0000", 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.784576+0000", 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.512 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.784577+0000", 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.2200000000000002e-07 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.784873+0000", 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029562499999999999 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.784886+0000", 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2514000000000001e-05 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.513 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.885029+0000", 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10014350599999999 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.985226+0000", 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100196673 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:10.985240+0000", 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3954999999999999e-05 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.514 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.088327+0000", 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103087603 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.088497+0000", 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00016963699999999999 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:987 3.16s0 3:69f31b93:::smithi18128752-39:head [delete] snapc 26=[26,25,24,22,20,1f,1e,1d] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e105)", 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:17.141321+0000", 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 1.2812740329999999, 2024-04-26T01:25:18.515 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39382061299999999, 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 987 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.141321+0000", 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.516 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.141321+0000", 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.141323+0000", 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.6070000000000002e-06 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.141336+0000", 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2536e-05 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.517 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.141368+0000", 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1622999999999999e-05 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.141383+0000", 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.562e-05 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.330434+0000", 2024-04-26T01:25:18.518 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.189051154 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.330611+0000", 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017631600000000001 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.331885+0000", 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0012742980000000001 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.334712+0000", 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0028272380000000001 2024-04-26T01:25:18.519 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.534911+0000", 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.200199179 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:17.535142+0000", 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023004199999999999 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.520 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:18.522 INFO:tasks.rados.rados.0.smithi181.stdout:396: finishing copy_from racing read to smithi18128752-14 2024-04-26T01:25:18.523 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-26T01:25:18.524 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:18.527+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:18.580 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing write tid 1 to smithi18128752-12 2024-04-26T01:25:18.580 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing write tid 2 to smithi18128752-12 2024-04-26T01:25:18.580 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing write tid 3 to smithi18128752-12 2024-04-26T01:25:18.581 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing write tid 4 to smithi18128752-12 2024-04-26T01:25:18.581 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing write tid 5 to smithi18128752-12 2024-04-26T01:25:18.610 INFO:tasks.rados.rados.0.smithi181.stdout:394: finishing rollback tid 0 to smithi18128752-50 2024-04-26T01:25:18.623 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:18.627+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:18.681 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing write tid 6 to smithi18128752-12 2024-04-26T01:25:18.681 INFO:tasks.rados.rados.0.smithi181.stdout:396: finishing copy_from to smithi18128752-14 2024-04-26T01:25:18.681 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 14 v 59 (ObjNum 92 snap 17 seq_num 92) dirty exists 2024-04-26T01:25:18.681 INFO:tasks.rados.rados.0.smithi181.stdout:396: done (5 left) 2024-04-26T01:25:18.681 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 5 2024-04-26T01:25:18.689 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "MOSDECSubOpWrite(3.ds6 106/63 ECSubWrite(tid=70, reqid=client.4589.0:998, at_version=0'0, trim_to=0'0, roll_forward_to=106'85))", 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:18.687415+0000", 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 0.0080982580000000005, 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0081178569999999992, 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "queued for pg", 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.690 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:18.687415+0000", 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:18.687415+0000", 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:18.687416+0000", 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.466e-06 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.691 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:18.687433+0000", 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6555e-05 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:18.687457+0000", 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3833e-05 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:18.687470+0000", 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2982e-05 2024-04-26T01:25:18.692 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.693 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:18.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.693 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:18.693 INFO:teuthology.orchestra.run.smithi037.stdout: ], 2024-04-26T01:25:18.693 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 1 2024-04-26T01:25:18.693 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:18.704 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-26T01:25:18.710 INFO:tasks.rados.rados.0.smithi181.stdout:394: finishing rollback tid 1 to smithi18128752-50 2024-04-26T01:25:18.711 INFO:tasks.rados.rados.0.smithi181.stdout:394: finishing rollback tid 2 to smithi18128752-50 2024-04-26T01:25:18.711 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 111 (ObjNum 128 snap 32 seq_num 128) dirty exists 2024-04-26T01:25:18.711 INFO:tasks.rados.rados.0.smithi181.stdout:394: done (4 left) 2024-04-26T01:25:18.711 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 4 2024-04-26T01:25:18.724 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:18.726+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:18.808 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T01:25:18.811 INFO:tasks.rados.rados.0.smithi181.stdout:393: finishing rollback tid 0 to smithi18128752-40 2024-04-26T01:25:18.825 INFO:tasks.ceph.osd.6.smithi071.stderr:2024-04-26T01:25:18.826+0000 7fb279eb9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25066) UID: 0 2024-04-26T01:25:18.865 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:18.865 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:18.865 INFO:teuthology.orchestra.run.smithi037.stdout: "complaint_time": 30, 2024-04-26T01:25:18.865 INFO:teuthology.orchestra.run.smithi037.stdout: "num_blocked_ops": 0 2024-04-26T01:25:18.865 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:18.879 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-26T01:25:18.912 INFO:tasks.rados.rados.0.smithi181.stdout:393: finishing rollback tid 1 to smithi18128752-40 2024-04-26T01:25:18.912 INFO:tasks.rados.rados.0.smithi181.stdout:393: finishing rollback tid 2 to smithi18128752-40 2024-04-26T01:25:18.913 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 179 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-04-26T01:25:18.913 INFO:tasks.rados.rados.0.smithi181.stdout:393: done (3 left) 2024-04-26T01:25:18.913 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 3 2024-04-26T01:25:18.913 INFO:tasks.rados.rados.0.smithi181.stdout:397: finishing write tid 1 to smithi18128752-19 2024-04-26T01:25:18.926 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:18.926+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:18.960 INFO:teuthology.orchestra.run.smithi037.stdout:osd.0: {} 2024-04-26T01:25:18.960 INFO:teuthology.orchestra.run.smithi037.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T01:25:18.981 INFO:teuthology.orchestra.run.smithi037.stdout:osd.1: {} 2024-04-26T01:25:18.981 INFO:teuthology.orchestra.run.smithi037.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T01:25:18.997 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "size": 20, 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 600, 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [ 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:123 3.ds0 3:b6ed1d05:::smithi18128752-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 73.507107778999995, 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.401407085, 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:18.998 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 123 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:18.999 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493544+0000", 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493545+0000", 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.5299999999999999e-07 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493546+0000", 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.7039999999999999e-06 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.000 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493552+0000", 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.5119999999999999e-06 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.493556+0000", 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2679999999999996e-06 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697434+0000", 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20387831000000001 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.001 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.697559+0000", 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012467700000000001 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087644+0000", 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.39008522000000001 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.002 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.087733+0000", 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.9191999999999996e-05 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.894951+0000", 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80721774899999998 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:138 3.1s0 3:8830b48d:::smithi18128752-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:25:19.003 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 73.221229973999996, 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71477871400000004, 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 138 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.004 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779422+0000", 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779423+0000", 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.47e-07 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779429+0000", 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6400000000000002e-06 2024-04-26T01:25:19.005 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779443+0000", 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4392999999999999e-05 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.779456+0000", 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2643000000000001e-05 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.006 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784323+0000", 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0048675369999999999 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:05.784379+0000", 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.6314000000000003e-05 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187338+0000", 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.40295840700000002 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.007 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.187417+0000", 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8898000000000003e-05 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.494201+0000", 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30678393500000001 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:141 3.ds0 3:b9916085:::smithi18128752-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-04-26T01:25:19.008 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 72.897758906000007, 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.99325914900000001, 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 141 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:19.009 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102893+0000", 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7500000000000001e-07 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.010 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102896+0000", 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3209999999999999e-06 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102900+0000", 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9369999999999997e-06 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.102903+0000", 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.3809999999999999e-06 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.011 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.291929+0000", 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.189026048 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.292027+0000", 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.8172999999999997e-05 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794279+0000", 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50225164799999999 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.012 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:06.794416+0000", 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00013710400000000001 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:07.096152+0000", 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.30173626199999998 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:325 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 60.639326336000003, 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70936785199999997, 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 325 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.014 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361326+0000", 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361328+0000", 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.3580000000000001e-06 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.015 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361338+0000", 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.628e-06 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361360+0000", 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2506000000000001e-05 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361374+0000", 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4072e-05 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.016 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454039+0000", 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.092664721000000005 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.454298+0000", 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00025867499999999999 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.455951+0000", 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0016533489999999999 2024-04-26T01:25:19.017 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462924+0000", 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0069735650000000001 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070553+0000", 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60762835599999998 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.018 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070693+0000", 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000140622 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:326 3.7s0 3:e977e128:::smithi18128752-21:head [rollback 4] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 60.63925777, 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70968584099999998, 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.019 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 326 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.020 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361394+0000", 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361395+0000", 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.06e-07 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361398+0000", 2024-04-26T01:25:19.021 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.1379999999999999e-06 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361408+0000", 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.046e-05 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361414+0000", 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8459999999999996e-06 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.022 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458365+0000", 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.096950878000000004 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.458663+0000", 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000297788 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.460203+0000", 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0015404030000000001 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.023 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.463428+0000", 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032243850000000002 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.070927+0000", 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.607499821 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071080+0000", 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000152516 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.024 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:327 3.7s0 3:e977e128:::smithi18128752-21:head [append 0~0] snapc 7=[7,4] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 60.639229944, 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71007962499999999, 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 327 2024-04-26T01:25:19.025 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.026 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361422+0000", 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6600000000000002e-07 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361425+0000", 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.2740000000000002e-06 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361431+0000", 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.4280000000000001e-06 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.027 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361436+0000", 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.994e-06 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461250+0000", 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099813814000000001 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.461584+0000", 2024-04-26T01:25:19.028 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000334521 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.462509+0000", 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00092412699999999996 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.565364+0000", 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.102855424 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.029 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071318+0000", 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50595366500000005 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.071502+0000", 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000183912 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:328 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:19.030 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 60.639209540000003, 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81248642000000004, 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 328 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:25:19.031 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361442+0000", 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361443+0000", 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6199999999999998e-07 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.032 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361447+0000", 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.262e-06 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361454+0000", 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.8739999999999997e-06 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361459+0000", 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.8040000000000001e-06 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.033 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563609+0000", 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20214983 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.563788+0000", 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017973199999999999 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.173929+0000", 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.610140456 2024-04-26T01:25:19.034 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:329 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 60.639186944000002, 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81314448500000003, 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.035 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 329 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.036 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361465+0000", 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2399999999999999e-07 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361469+0000", 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.2820000000000001e-06 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361475+0000", 2024-04-26T01:25:19.037 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.3509999999999998e-06 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361480+0000", 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.6959999999999996e-06 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564389+0000", 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.202908808 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.038 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.564593+0000", 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000204255 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.174609+0000", 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.61001666899999996 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:330 3.7s0 3:e9e07e85:::smithi18128752-17:head [read 0~1212416,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:25:19.039 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 60.639166433, 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.81407258199999999, 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 330 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:25:19.040 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361485+0000", 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361486+0000", 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.0999999999999999e-07 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.041 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361489+0000", 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.7240000000000001e-06 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361495+0000", 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.2740000000000004e-06 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.361500+0000", 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.9400000000000001e-06 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.042 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.866935+0000", 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50543468599999997 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:18.867233+0000", 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00029877499999999998 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:19.175558+0000", 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.308324773 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.043 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:434 3.7s0 3:e9e07e85:::smithi18128752-17:head [copy-from ver 29 in=54b] snapc d=[d,c,8,4] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 50.221999754000002, 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1051031140000001, 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.044 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 434 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778652+0000", 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.045 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778655+0000", 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8770000000000002e-06 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778667+0000", 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.1888999999999999e-05 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778697+0000", 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0205999999999998e-05 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.046 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778714+0000", 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.6467999999999999e-05 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878928+0000", 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100213993 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "op must wait for map", 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878940+0000", 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.2724999999999999e-05 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.047 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882181+0000", 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0032411419999999998 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882351+0000", 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000169371 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.678907+0000", 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.79655591199999998 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.048 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.882986+0000", 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.204079063 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883081+0000", 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 9.5285000000000007e-05 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.883755+0000", 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00067418299999999997 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.049 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:435 3.7s0 3:e9e07e85:::smithi18128752-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 50.221904096999999, 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.003729404, 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.050 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 435 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778748+0000", 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.051 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778749+0000", 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.223e-06 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778755+0000", 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.8279999999999999e-06 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.778773+0000", 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.8289000000000001e-05 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.052 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.878906+0000", 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10013274699999999 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879035+0000", 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00012864199999999999 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting_for_map not empty", 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.879049+0000", 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.4137e-05 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.053 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882650+0000", 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0036017390000000001 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for blocked object", 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:28.882716+0000", 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5129000000000006e-05 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782226+0000", 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.89951035899999998 2024-04-26T01:25:19.054 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782341+0000", 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000115492 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:29.782477+0000", 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000135819 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.055 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:477 3.bs0 3:d65fea44:::smithi18128752-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e61)", 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 46.108072458000002, 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.91386205600000003, 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 477 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.056 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892579+0000", 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892580+0000", 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.7999999999999996e-07 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.057 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892582+0000", 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9630000000000001e-06 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892589+0000", 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.7959999999999998e-06 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.892593+0000", 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2509999999999996e-06 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.058 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992304+0000", 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099710909 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:32.992356+0000", 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.2021000000000002e-05 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.402928+0000", 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.41057238600000001 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.059 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.403133+0000", 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00020520099999999999 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:33.806441+0000", 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.403308049 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:537 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~704512] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:19.060 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.173484309999999, 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77667111300000002, 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 537 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.061 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827168+0000", 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.827172+0000", 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9639999999999997e-06 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.062 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.828324+0000", 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011519550000000001 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.828352+0000", 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.8112000000000001e-05 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.928473+0000", 2024-04-26T01:25:19.063 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.100121081 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.994151+0000", 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.065677997000000002 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.994232+0000", 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1688999999999997e-05 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.995096+0000", 2024-04-26T01:25:19.064 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00086366299999999997 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997758+0000", 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0026614830000000001 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.603693+0000", 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60593560000000002 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.603839+0000", 2024-04-26T01:25:19.065 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00014556899999999999 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:539 3.1s0 3:865c3313:::smithi18128752-37:head [append 0~81920] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.071531114000003, 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77547855799999998, 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.066 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 539 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:19.067 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929121+0000", 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5400000000000001e-07 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929175+0000", 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.4033000000000003e-05 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929183+0000", 2024-04-26T01:25:19.068 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.7130000000000005e-06 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929187+0000", 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.2719999999999999e-06 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997790+0000", 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.068602814999999998 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.997917+0000", 2024-04-26T01:25:19.069 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000126783 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998505+0000", 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00058842300000000003 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999674+0000", 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0011685179999999999 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.070 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704445+0000", 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.704771335 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704599+0000", 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000154012 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:540 3.1s0 3:865c3313:::smithi18128752-37:head [setxattr _header (57) in=7b] snapc 11=[10,e] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:19.071 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.071458069999998, 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.77584385600000005, 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 540 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:19.072 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929194+0000", 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.9200000000000002e-07 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929197+0000", 2024-04-26T01:25:19.073 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.661e-06 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929201+0000", 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.194e-06 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929204+0000", 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4020000000000002e-06 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.074 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998891+0000", 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.069686039000000005 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.998997+0000", 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00010671 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999475+0000", 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00047769400000000002 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.075 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.999988+0000", 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051352600000000004 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.704829+0000", 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.70484043500000004 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.705038+0000", 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000208803 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.076 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:541 3.1s0 3:865c3313:::smithi18128752-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e67)", 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 40.071442801000003, 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0548109999999999, 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 541 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.077 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929209+0000", 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 2.9999999999999999e-07 2024-04-26T01:25:19.078 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929211+0000", 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.517e-06 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929217+0000", 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.0410000000000003e-06 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.079 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:38.929221+0000", 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.659e-06 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.028709+0000", 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.099488691000000004 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.028788+0000", 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.8268000000000003e-05 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.080 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.707449+0000", 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.67866140799999997 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.707686+0000", 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00023745600000000001 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:39.984020+0000", 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.27633365999999998 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.081 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:760 3.1s0 3:8830b48d:::smithi18128752-46:head [setxattr _header (57) in=7b] snapc 19=[19,18] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.225249+0000", 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.775402526000001, 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71216155800000003, 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 760 2024-04-26T01:25:19.082 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225249+0000", 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225249+0000", 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225251+0000", 2024-04-26T01:25:19.083 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.237e-06 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225257+0000", 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 6.5370000000000003e-06 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225270+0000", 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.3305999999999999e-05 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.084 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225279+0000", 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.5969999999999992e-06 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236485+0000", 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.011206317 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.236734+0000", 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000248206 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.085 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.237890+0000", 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.001156575 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.238966+0000", 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0010758689999999999 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.937219+0000", 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.69825322400000001 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.086 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.937411+0000", 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019169 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:761 3.1s0 3:8830b48d:::smithi18128752-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:24:56.225291+0000", 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 22.775360988999999, 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.71929514500000002, 2024-04-26T01:25:19.087 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 761 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225291+0000", 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.088 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225291+0000", 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225292+0000", 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 7.6700000000000003e-07 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225295+0000", 2024-04-26T01:25:19.089 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4759999999999998e-06 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225306+0000", 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.0883999999999999e-05 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.225314+0000", 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 8.1799999999999996e-06 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.090 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.330700+0000", 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10538581800000001 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.330964+0000", 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00026375200000000002 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.938144+0000", 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.60718067099999995 2024-04-26T01:25:19.091 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.938316+0000", 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00017196600000000001 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:24:56.944586+0000", 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.0062696310000000003 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.092 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:891 3.1s0 3:87197ca9:::smithi18128752-6:head [copy-from ver 74] snapc 22=[22,21,20,1f,1e,1d] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:11.049342+0000", 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.9513103620000001, 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80762111999999997, 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 891 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.093 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.049342+0000", 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.049342+0000", 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.094 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.049345+0000", 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.0630000000000002e-06 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.049360+0000", 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.525e-05 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.049394+0000", 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.4403999999999998e-05 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.095 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.149487+0000", 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10009235800000001 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.149602+0000", 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000115653 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.150115+0000", 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00051235400000000002 2024-04-26T01:25:19.096 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started", 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.640086+0000", 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.48997118499999998 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_committed", 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.752992+0000", 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.11290612999999999 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "commit_sent", 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.856765+0000", 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.103772604 2024-04-26T01:25:19.097 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.856963+0000", 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00019811899999999999 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "description": "osd_op(client.4589.0:898 3.1s0 3:865c3313:::smithi18128752-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "initiated_at": "2024-04-26T01:25:11.151703+0000", 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "age": 7.8489492839999997, 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.80749965800000001, 2024-04-26T01:25:19.098 INFO:teuthology.orchestra.run.smithi037.stdout: "continuous": false, 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "type_data": { 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "flag_point": "started", 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "client_info": { 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4589", 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.181:0/4253793648", 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 898 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "events": [ 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated", 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.151703+0000", 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.099 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "header_read", 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.151703+0000", 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled", 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.151703+0000", 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 3.7399999999999999e-07 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "all_read", 2024-04-26T01:25:19.100 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.151705+0000", 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 1.9300000000000002e-06 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "dispatched", 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.151710+0000", 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 5.3029999999999999e-06 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg", 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.151715+0000", 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 4.4100000000000001e-06 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.101 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.354377+0000", 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.20266213999999999 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "waiting for rw locks", 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.354497+0000", 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.000120259 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg", 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.855371+0000", 2024-04-26T01:25:19.102 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.50087411900000001 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "started", 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.855642+0000", 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.00027035299999999998 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: }, 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: { 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done", 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2024-04-26T01:25:11.959202+0000", 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: "duration": 0.10356077 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.103 INFO:teuthology.orchestra.run.smithi037.stdout: } 2024-04-26T01:25:19.104 INFO:teuthology.orchestra.run.smithi037.stdout: ] 2024-04-26T01:25:19.104 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:19.105 INFO:teuthology.orchestra.run.smithi037.stdout:osd.2: {} 2024-04-26T01:25:19.105 INFO:teuthology.orchestra.run.smithi037.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T01:25:19.106 INFO:tasks.rados.rados.0.smithi181.stdout:397: finishing write tid 2 to smithi18128752-19 2024-04-26T01:25:19.106 INFO:tasks.rados.rados.0.smithi181.stdout:397: finishing write tid 3 to smithi18128752-19 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:397: finishing write tid 4 to smithi18128752-19 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing write tid 7 to smithi18128752-12 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: oid 12 updating version 0 to 49 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: oid 12 version 49 is already newer than 45 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: oid 12 version 49 is already newer than 47 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: oid 12 updating version 49 to 50 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: oid 12 version 50 is already newer than 48 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: oid 12 version 50 is already newer than 46 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 50 (ObjNum 143 snap 38 seq_num 143) dirty exists 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: left oid 12 (ObjNum 143 snap 38 seq_num 143) 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout:398: done (2 left) 2024-04-26T01:25:19.107 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 2 2024-04-26T01:25:19.108 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-04-26T01:25:19.108 INFO:teuthology.orchestra.run.smithi037.stdout:osd.3: {} 2024-04-26T01:25:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T01:25:19.109 INFO:teuthology.orchestra.run.smithi037.stdout:osd.4: {} 2024-04-26T01:25:19.109 INFO:teuthology.orchestra.run.smithi037.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T01:25:19.110 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:25:19.110 INFO:teuthology.orchestra.run.smithi037.stderr:set pool 3 pg_num to 36 2024-04-26T01:25:19.110 INFO:teuthology.orchestra.run.smithi037.stdout:osd.5: {} 2024-04-26T01:25:19.111 INFO:teuthology.orchestra.run.smithi037.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T01:25:19.111 INFO:tasks.ceph.osd.13.smithi045.stderr:2024-04-26T01:25:19.106+0000 7f04f5d16640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 24941) UID: 0 2024-04-26T01:25:19.113 INFO:tasks.rados.rados.0.smithi181.stdout:397: finishing write tid 5 to smithi18128752-19 2024-04-26T01:25:19.192 INFO:teuthology.orchestra.run.smithi037.stdout:osd.6: {} 2024-04-26T01:25:19.192 INFO:teuthology.orchestra.run.smithi037.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T01:25:19.206 INFO:tasks.ceph.osd.8.smithi037.stderr:2024-04-26T01:25:19.207+0000 7f982ebe8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 25064) UID: 0 2024-04-26T01:25:19.219 INFO:tasks.rados.rados.0.smithi181.stdout:397: finishing write tid 6 to smithi18128752-19 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:397: oid 19 updating version 0 to 180 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:397: oid 19 updating version 180 to 181 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:397: oid 19 updating version 181 to 183 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:397: oid 19 updating version 183 to 184 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:397: oid 19 version 184 is already newer than 182 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 184 (ObjNum 142 snap 38 seq_num 142) dirty exists 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:397: left oid 19 (ObjNum 142 snap 38 seq_num 142) 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout:397: done (1 left) 2024-04-26T01:25:19.220 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 1 2024-04-26T01:25:19.267 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:25:19.267 INFO:teuthology.orchestra.run.smithi037.stdout: "ops": [], 2024-04-26T01:25:19.267 INFO:teuthology.orchestra.run.smithi037.stdout: "num_ops": 0 2024-04-26T01:25:19.267 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:25:19.281 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-04-26T01:25:19.284 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing rollback tid 1 to smithi18128752-43 2024-04-26T01:25:19.307 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-04-26T01:25:19.308+0000 7f8642c3a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25318) UID: 0 2024-04-26T01:25:19.315 INFO:teuthology.orchestra.run.smithi037.stdout:osd.7: {} 2024-04-26T01:25:19.315 INFO:teuthology.orchestra.run.smithi037.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T01:25:19.380 INFO:teuthology.orchestra.run.smithi037.stdout:osd.8: {} 2024-04-26T01:25:19.381 INFO:teuthology.orchestra.run.smithi037.stderr:osd.8: osd_enable_op_tracker = '' 2024-04-26T01:25:19.385 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing rollback tid 2 to smithi18128752-43 2024-04-26T01:25:19.385 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 143 (ObjNum 100 snap 17 seq_num 100) dirty exists 2024-04-26T01:25:19.386 INFO:tasks.rados.rados.0.smithi181.stdout:352: done (0 left) 2024-04-26T01:25:19.394 INFO:tasks.rados.rados.0.smithi181.stderr:0 errors. 2024-04-26T01:25:19.395 INFO:tasks.rados.rados.0.smithi181.stderr: 2024-04-26T01:25:19.407 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:25:19.408 INFO:teuthology.orchestra.run.smithi037.stdout:osd.9: {} 2024-04-26T01:25:19.409 INFO:teuthology.orchestra.run.smithi037.stderr:osd.9: osd_enable_op_tracker = '' 2024-04-26T01:25:19.410 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:19.410+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:19.436 INFO:teuthology.orchestra.run.smithi037.stdout:osd.10: {} 2024-04-26T01:25:19.437 INFO:teuthology.orchestra.run.smithi037.stderr:osd.10: osd_enable_op_tracker = '' 2024-04-26T01:25:19.440 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:19.441 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:19.456 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-04-26T01:25:19.465 INFO:teuthology.orchestra.run.smithi037.stdout:osd.11: {} 2024-04-26T01:25:19.465 INFO:teuthology.orchestra.run.smithi037.stderr:osd.11: osd_enable_op_tracker = '' 2024-04-26T01:25:19.484 INFO:teuthology.orchestra.run.smithi037.stdout:osd.12: {} 2024-04-26T01:25:19.484 INFO:teuthology.orchestra.run.smithi037.stderr:osd.12: osd_enable_op_tracker = '' 2024-04-26T01:25:19.507 INFO:teuthology.orchestra.run.smithi037.stdout:osd.13: {} 2024-04-26T01:25:19.507 INFO:teuthology.orchestra.run.smithi037.stderr:osd.13: osd_enable_op_tracker = '' 2024-04-26T01:25:19.509 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:19.511+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:19.528 INFO:teuthology.orchestra.run.smithi037.stdout:osd.14: {} 2024-04-26T01:25:19.528 INFO:teuthology.orchestra.run.smithi037.stderr:osd.14: osd_enable_op_tracker = '' 2024-04-26T01:25:19.549 INFO:teuthology.orchestra.run.smithi037.stdout:osd.15: {} 2024-04-26T01:25:19.550 INFO:teuthology.orchestra.run.smithi037.stderr:osd.15: osd_enable_op_tracker = '' 2024-04-26T01:25:19.583 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:19.583 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:19.598 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-04-26T01:25:19.611 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:19.612+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:19.709 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:19.709 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":107,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:25:19.067417+0000","last_up_change":"2024-04-26T01:23:47.867650+0000","last_in_change":"2024-04-26T01:25:01.794306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:23:47.920762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99490642547607422,"average_primary_affinity":0.99000000953674316,"average_primary_affinity_weighted":0.99000000953674316}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:23:50.025173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99490642547607422,"average_primary_affinity":0.99000000953674316,"average_primary_affinity_weighted":0.99000000953674316}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-04-26T01:23:58.994788+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":16,"pg_placement_num_target":36,"pg_num_target":36,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"102","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6819","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6820","nonce":906916438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6824","nonce":906916438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6829","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6831","nonce":906916438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6827","nonce":906916438}]},"public_addr":"172.21.15.37:6820/906916438","cluster_addr":"172.21.15.37:6824/906916438","heartbeat_back_addr":"172.21.15.37:6831/906916438","heartbeat_front_addr":"172.21.15.37:6827/906916438","state":["exists","up"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6813","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6816","nonce":1630954663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6823","nonce":1630954663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6831","nonce":1630954663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6828","nonce":1630954663}]},"public_addr":"172.21.15.45:6816/1630954663","cluster_addr":"172.21.15.45:6823/1630954663","heartbeat_back_addr":"172.21.15.45:6831/1630954663","heartbeat_front_addr":"172.21.15.45:6828/1630954663","state":["exists","up"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6801","nonce":2961312835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6803","nonce":2961312835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6807","nonce":2961312835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6805","nonce":2961312835}]},"public_addr":"172.21.15.71:6801/2961312835","cluster_addr":"172.21.15.71:6803/2961312835","heartbeat_back_addr":"172.21.15.71:6807/2961312835","heartbeat_front_addr":"172.21.15.71:6805/2961312835","state":["exists","up"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6801","nonce":1026435117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6806","nonce":1026435117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6821","nonce":1026435117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6814","nonce":1026435117}]},"public_addr":"172.21.15.181:6801/1026435117","cluster_addr":"172.21.15.181:6806/1026435117","heartbeat_back_addr":"172.21.15.181:6821/1026435117","heartbeat_front_addr":"172.21.15.181:6814/1026435117","state":["exists","up"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6803","nonce":1575437742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6805","nonce":1575437742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6809","nonce":1575437742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6807","nonce":1575437742}]},"public_addr":"172.21.15.37:6803/1575437742","cluster_addr":"172.21.15.37:6805/1575437742","heartbeat_back_addr":"172.21.15.37:6809/1575437742","heartbeat_front_addr":"172.21.15.37:6807/1575437742","state":["exists","up"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":1,"in":1,"weight":1,"primary_affinity":0.9185028076171875,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6801","nonce":3759230996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6803","nonce":3759230996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6807","nonce":3759230996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6805","nonce":3759230996}]},"public_addr":"172.21.15.45:6801/3759230996","cluster_addr":"172.21.15.45:6803/3759230996","heartbeat_back_addr":"172.21.15.45:6807/3759230996","heartbeat_front_addr":"172.21.15.45:6805/3759230996","state":["exists","up"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6809","nonce":76180415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6811","nonce":76180415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6815","nonce":76180415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6813","nonce":76180415}]},"public_addr":"172.21.15.71:6809/76180415","cluster_addr":"172.21.15.71:6811/76180415","heartbeat_back_addr":"172.21.15.71:6815/76180415","heartbeat_front_addr":"172.21.15.71:6813/76180415","state":["exists","up"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6807","nonce":4043929361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6812","nonce":4043929361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6823","nonce":4043929361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6820","nonce":4043929361}]},"public_addr":"172.21.15.181:6807/4043929361","cluster_addr":"172.21.15.181:6812/4043929361","heartbeat_back_addr":"172.21.15.181:6823/4043929361","heartbeat_front_addr":"172.21.15.181:6820/4043929361","state":["exists","up"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6811","nonce":4094482142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6813","nonce":4094482142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6817","nonce":4094482142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6815","nonce":4094482142}]},"public_addr":"172.21.15.37:6811/4094482142","cluster_addr":"172.21.15.37:6813/4094482142","heartbeat_back_addr":"172.21.15.37:6817/4094482142","heartbeat_front_addr":"172.21.15.37:6815/4094482142","state":["exists","up"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6810","nonce":3448396888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6815","nonce":3448396888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6824","nonce":3448396888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6817","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6819","nonce":3448396888}]},"public_addr":"172.21.15.45:6810/3448396888","cluster_addr":"172.21.15.45:6815/3448396888","heartbeat_back_addr":"172.21.15.45:6824/3448396888","heartbeat_front_addr":"172.21.15.45:6819/3448396888","state":["exists","up"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6817","nonce":896289472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6819","nonce":896289472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6823","nonce":896289472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6821","nonce":896289472}]},"public_addr":"172.21.15.71:6817/896289472","cluster_addr":"172.21.15.71:6819/896289472","heartbeat_back_addr":"172.21.15.71:6823/896289472","heartbeat_front_addr":"172.21.15.71:6821/896289472","state":["exists","up"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6825","nonce":647568313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6827","nonce":647568313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6831","nonce":647568313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6829","nonce":647568313}]},"public_addr":"172.21.15.181:6825/647568313","cluster_addr":"172.21.15.181:6827/647568313","heartbeat_back_addr":"172.21.15.181:6831/647568313","heartbeat_front_addr":"172.21.15.181:6829/647568313","state":["exists","up"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6821","nonce":3336489453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6825","nonce":3336489453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6833","nonce":3336489453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6830","nonce":3336489453}]},"public_addr":"172.21.15.37:6821/3336489453","cluster_addr":"172.21.15.37:6825/3336489453","heartbeat_back_addr":"172.21.15.37:6833/3336489453","heartbeat_front_addr":"172.21.15.37:6830/3336489453","state":["exists","up"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6811","nonce":1608953722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6818","nonce":1608953722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6827","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6829","nonce":1608953722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6821","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6825","nonce":1608953722}]},"public_addr":"172.21.15.45:6811/1608953722","cluster_addr":"172.21.15.45:6818/1608953722","heartbeat_back_addr":"172.21.15.45:6829/1608953722","heartbeat_front_addr":"172.21.15.45:6825/1608953722","state":["exists","up"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6825","nonce":3719375992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6827","nonce":3719375992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6831","nonce":3719375992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6829","nonce":3719375992}]},"public_addr":"172.21.15.71:6825/3719375992","cluster_addr":"172.21.15.71:6827/3719375992","heartbeat_back_addr":"172.21.15.71:6831/3719375992","heartbeat_front_addr":"172.21.15.71:6829/3719375992","state":["exists","up"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6805","nonce":2891128021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6811","nonce":2891128021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6817","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6819","nonce":2891128021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6815","nonce":2891128021}]},"public_addr":"172.21.15.181:6805/2891128021","cluster_addr":"172.21.15.181:6811/2891128021","heartbeat_back_addr":"172.21.15.181:6819/2891128021","heartbeat_front_addr":"172.21.15.181:6815/2891128021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.150989+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:45.701279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-04-26T01:24:55.015898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.670769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.579754+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:08.910977+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:49.055076+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:48.659623+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.515953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:14.187344+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:13.456713+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:02.015135+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.667171+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.495812+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:48.329599+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:47.498480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":25,"length":1},{"begin":27,"length":2},{"begin":33,"length":1},{"begin":35,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:25:19.712 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:19.714+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:19.730 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:25:19.760 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:19.761 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:19.776 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-04-26T01:25:19.813 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:19.814+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:19.914 INFO:tasks.ceph.osd.14.smithi071.stderr:2024-04-26T01:25:19.914+0000 7f3b1430e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 25070) UID: 0 2024-04-26T01:25:19.943 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:19.943 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:19.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-04-26T01:25:20.014 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-04-26T01:25:20.016+0000 7f1ff4a50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 25316) UID: 0 2024-04-26T01:25:20.051 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:20.052 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:25:20.070 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-26T01:25:19.843917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129483296,"num_objects":74,"num_object_clones":28,"num_object_copies":508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":6,"num_read":537,"num_read_kb":148512,"num_write":828,"num_write_kb":144964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":57848352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2024,"ondisk_log_size":2024,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":735172,"kb_used_data":263304,"kb_used_omap":23,"kb_used_meta":471656,"kb_avail":1414842428,"statfs":{"total":1449551462400,"available":1448798646272,"internally_reserved":0,"allocated":269623296,"data_stored":343586255,"data_compressed":3792813,"data_compressed_allocated":5197824,"data_compressed_original":11268096,"omap_allocated":23926,"internal_metadata":482976394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9977856,"num_objects":6,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":-43,"num_read_kb":4831,"num_write":-39,"num_write_kb":-18592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10,"num_bytes_recovered":-1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001436"},"pg_stats":[{"pgid":"3.19","version":"91'65","reported_seq":428,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.157279+0000","last_change":"2024-04-26T01:25:15.538612+0000","last_active":"2024-04-26T01:25:18.157279+0000","last_peered":"2024-04-26T01:25:18.157279+0000","last_clean":"2024-04-26T01:25:18.157279+0000","last_became_active":"2024-04-26T01:25:15.347044+0000","last_became_peered":"2024-04-26T01:25:15.347044+0000","last_unstale":"2024-04-26T01:25:18.157279+0000","last_undegraded":"2024-04-26T01:25:18.157279+0000","last_fullsized":"2024-04-26T01:25:18.157279+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10094185,"stat_sum":{"num_bytes":2859008,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":508,"num_write":12,"num_write_kb":3832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.18","version":"105'141","reported_seq":583,"reported_epoch":105,"state":"active+recovering+degraded","last_fresh":"2024-04-26T01:25:17.539125+0000","last_change":"2024-04-26T01:25:16.497881+0000","last_active":"2024-04-26T01:25:17.539125+0000","last_peered":"2024-04-26T01:25:17.539125+0000","last_clean":"2024-04-26T01:25:01.261092+0000","last_became_active":"2024-04-26T01:25:15.750526+0000","last_became_peered":"2024-04-26T01:25:15.750526+0000","last_unstale":"2024-04-26T01:25:17.539125+0000","last_undegraded":"2024-04-26T01:25:15.446828+0000","last_fullsized":"2024-04-26T01:25:17.539125+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11116544,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":26,"num_read_kb":13291,"num_write":58,"num_write_kb":13592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":7766016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,12,1,14,15,6],"acting":[5,0,12,1,14,15,6],"avail_no_missing":["5(0)","0(1)","1(2)","6(6)","9(3)","14(4)","15(5)"],"object_location_counts":[{"shards":"0(1),1(3),5(0),6(6),12(2),14(4),15(5)","objects":4},{"shards":"0(1),5(0),6(6),14(4),15(5)","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"100'109","reported_seq":499,"reported_epoch":106,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T01:25:18.212406+0000","last_change":"2024-04-26T01:25:17.657399+0000","last_active":"2024-04-26T01:25:18.212406+0000","last_peered":"2024-04-26T01:25:18.212406+0000","last_clean":"2024-04-26T01:25:18.212406+0000","last_became_active":"2024-04-26T01:25:15.762430+0000","last_became_peered":"2024-04-26T01:25:15.762430+0000","last_unstale":"2024-04-26T01:25:18.212406+0000","last_undegraded":"2024-04-26T01:25:18.212406+0000","last_fullsized":"2024-04-26T01:25:18.212406+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'103","last_scrub_stamp":"2024-04-26T01:25:08.533710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:08.533710+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0.101220037,"stat_sum":{"num_bytes":5914624,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":2816,"num_write":32,"num_write_kb":6608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.16","version":"105'44","reported_seq":463,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.534916+0000","last_change":"2024-04-26T01:25:15.662433+0000","last_active":"2024-04-26T01:25:17.534916+0000","last_peered":"2024-04-26T01:25:17.534916+0000","last_clean":"2024-04-26T01:25:17.534916+0000","last_became_active":"2024-04-26T01:25:15.555647+0000","last_became_peered":"2024-04-26T01:25:15.555647+0000","last_unstale":"2024-04-26T01:25:17.534916+0000","last_undegraded":"2024-04-26T01:25:17.534916+0000","last_fullsized":"2024-04-26T01:25:17.534916+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:22.536799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.103614964,"stat_sum":{"num_bytes":2318336,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":8,"num_read_kb":1324,"num_write":15,"num_write_kb":3904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.15","version":"0'0","reported_seq":380,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.128681+0000","last_change":"2024-04-26T01:25:15.554363+0000","last_active":"2024-04-26T01:25:17.128681+0000","last_peered":"2024-04-26T01:25:17.128681+0000","last_clean":"2024-04-26T01:25:17.128681+0000","last_became_active":"2024-04-26T01:25:15.450430+0000","last_became_peered":"2024-04-26T01:25:15.450430+0000","last_unstale":"2024-04-26T01:25:17.128681+0000","last_undegraded":"2024-04-26T01:25:17.128681+0000","last_fullsized":"2024-04-26T01:25:17.128681+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:22.818125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101743263,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.14","version":"104'78","reported_seq":487,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.069257+0000","last_change":"2024-04-26T01:25:17.601885+0000","last_active":"2024-04-26T01:25:18.069257+0000","last_peered":"2024-04-26T01:25:18.069257+0000","last_clean":"2024-04-26T01:25:18.069257+0000","last_became_active":"2024-04-26T01:25:15.853290+0000","last_became_peered":"2024-04-26T01:25:15.853290+0000","last_unstale":"2024-04-26T01:25:18.069257+0000","last_undegraded":"2024-04-26T01:25:18.069257+0000","last_fullsized":"2024-04-26T01:25:18.069257+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'78","last_scrub_stamp":"2024-04-26T01:25:17.601441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:17.601441+0000","objects_scrubbed":5,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:42.627673+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0.40770353199999998,"stat_sum":{"num_bytes":18317312,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":167,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.128684+0000","last_change":"2024-04-26T01:23:51.916223+0000","last_active":"2024-04-26T01:25:17.128684+0000","last_peered":"2024-04-26T01:25:17.128684+0000","last_clean":"2024-04-26T01:25:17.128684+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:17.128684+0000","last_undegraded":"2024-04-26T01:25:17.128684+0000","last_fullsized":"2024-04-26T01:25:17.128684+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:17.452696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"72'37","reported_seq":456,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.130223+0000","last_change":"2024-04-26T01:25:15.356119+0000","last_active":"2024-04-26T01:25:17.130223+0000","last_peered":"2024-04-26T01:25:17.130223+0000","last_clean":"2024-04-26T01:25:17.130223+0000","last_became_active":"2024-04-26T01:25:15.352396+0000","last_became_peered":"2024-04-26T01:25:15.352396+0000","last_unstale":"2024-04-26T01:25:17.130223+0000","last_undegraded":"2024-04-26T01:25:17.130223+0000","last_fullsized":"2024-04-26T01:25:17.130223+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001905265,"stat_sum":{"num_bytes":57344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":1322,"num_write":9,"num_write_kb":1696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,1,15,12,7,9],"acting":[0,13,1,15,12,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:19.031240+0000","last_change":"2024-04-26T01:25:19.031279+0000","last_active":"2024-04-26T01:25:19.031240+0000","last_peered":"2024-04-26T01:25:19.031240+0000","last_clean":"2024-04-26T01:25:19.031240+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:19.031240+0000","last_undegraded":"2024-04-26T01:25:19.031240+0000","last_fullsized":"2024-04-26T01:25:19.031240+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:19.031090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:19.031090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.870416+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"100'109","reported_seq":495,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.080297+0000","last_change":"2024-04-26T01:25:15.658369+0000","last_active":"2024-04-26T01:25:18.080297+0000","last_peered":"2024-04-26T01:25:18.080297+0000","last_clean":"2024-04-26T01:25:18.080297+0000","last_became_active":"2024-04-26T01:25:15.453119+0000","last_became_peered":"2024-04-26T01:25:15.453119+0000","last_unstale":"2024-04-26T01:25:18.080297+0000","last_undegraded":"2024-04-26T01:25:18.080297+0000","last_fullsized":"2024-04-26T01:25:18.080297+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'103","last_scrub_stamp":"2024-04-26T01:25:08.533710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:08.533710+0000","objects_scrubbed":4,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0.102987312,"stat_sum":{"num_bytes":5914624,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":2816,"num_write":32,"num_write_kb":6608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,0,8,5,11],"acting":[4,6,9,0,8,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":166,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.137547+0000","last_change":"2024-04-26T01:23:52.015219+0000","last_active":"2024-04-26T01:25:17.137547+0000","last_peered":"2024-04-26T01:25:17.137547+0000","last_clean":"2024-04-26T01:25:17.137547+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:17.137547+0000","last_undegraded":"2024-04-26T01:25:17.137547+0000","last_fullsized":"2024-04-26T01:25:17.137547+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:18.338853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"100'72","reported_seq":477,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.437542+0000","last_change":"2024-04-26T01:25:18.437617+0000","last_active":"2024-04-26T01:25:18.437542+0000","last_peered":"2024-04-26T01:25:18.437542+0000","last_clean":"2024-04-26T01:25:18.437542+0000","last_became_active":"2024-04-26T01:25:15.445723+0000","last_became_peered":"2024-04-26T01:25:15.445723+0000","last_unstale":"2024-04-26T01:25:18.437542+0000","last_undegraded":"2024-04-26T01:25:18.437542+0000","last_fullsized":"2024-04-26T01:25:18.437542+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'72","last_scrub_stamp":"2024-04-26T01:25:18.437402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:18.437402+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:29.037085+0000","scrub_duration":535,"objects_trimmed":0,"snaptrim_duration":0.50580246500000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4,1,0,14,13],"acting":[8,9,4,1,0,14,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.079958+0000","last_change":"2024-04-26T01:25:10.080365+0000","last_active":"2024-04-26T01:25:18.079958+0000","last_peered":"2024-04-26T01:25:18.079958+0000","last_clean":"2024-04-26T01:25:18.079958+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:18.079958+0000","last_undegraded":"2024-04-26T01:25:18.079958+0000","last_fullsized":"2024-04-26T01:25:18.079958+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:10.080096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:10.080096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.198292+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":380,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.128742+0000","last_change":"2024-04-26T01:25:15.249798+0000","last_active":"2024-04-26T01:25:17.128742+0000","last_peered":"2024-04-26T01:25:17.128742+0000","last_clean":"2024-04-26T01:25:17.128742+0000","last_became_active":"2024-04-26T01:25:15.148668+0000","last_became_peered":"2024-04-26T01:25:15.148668+0000","last_unstale":"2024-04-26T01:25:17.128742+0000","last_undegraded":"2024-04-26T01:25:17.128742+0000","last_fullsized":"2024-04-26T01:25:17.128742+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016243449999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,4,13,7,0,6],"acting":[15,8,4,13,7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.3","version":"100'15","reported_seq":418,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.113855+0000","last_change":"2024-04-26T01:25:15.765940+0000","last_active":"2024-04-26T01:25:18.113855+0000","last_peered":"2024-04-26T01:25:18.113855+0000","last_clean":"2024-04-26T01:25:18.113855+0000","last_became_active":"2024-04-26T01:25:15.456024+0000","last_became_peered":"2024-04-26T01:25:15.456024+0000","last_unstale":"2024-04-26T01:25:18.113855+0000","last_undegraded":"2024-04-26T01:25:18.113855+0000","last_fullsized":"2024-04-26T01:25:18.113855+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.109821049,"stat_sum":{"num_bytes":1490944,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":8363,"num_write":12,"num_write_kb":1760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,6,5,3],"acting":[13,7,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":179,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.294950+0000","last_change":"2024-04-26T01:25:02.920443+0000","last_active":"2024-04-26T01:25:17.294950+0000","last_peered":"2024-04-26T01:25:17.294950+0000","last_clean":"2024-04-26T01:25:17.294950+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:17.294950+0000","last_undegraded":"2024-04-26T01:25:17.294950+0000","last_fullsized":"2024-04-26T01:25:17.294950+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:19.351681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"105'175","reported_seq":569,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.254681+0000","last_change":"2024-04-26T01:25:15.542184+0000","last_active":"2024-04-26T01:25:18.254681+0000","last_peered":"2024-04-26T01:25:18.254681+0000","last_clean":"2024-04-26T01:25:18.254681+0000","last_became_active":"2024-04-26T01:25:15.439145+0000","last_became_peered":"2024-04-26T01:25:15.439145+0000","last_unstale":"2024-04-26T01:25:18.254681+0000","last_undegraded":"2024-04-26T01:25:18.254681+0000","last_fullsized":"2024-04-26T01:25:18.254681+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083421299999999999,"stat_sum":{"num_bytes":3940352,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":27,"num_read_kb":11939,"num_write":54,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1875968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.113141+0000","last_change":"2024-04-26T01:23:51.906525+0000","last_active":"2024-04-26T01:25:18.113141+0000","last_peered":"2024-04-26T01:25:18.113141+0000","last_clean":"2024-04-26T01:25:18.113141+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:18.113141+0000","last_undegraded":"2024-04-26T01:25:18.113141+0000","last_fullsized":"2024-04-26T01:25:18.113141+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.1","version":"103'67","reported_seq":486,"reported_epoch":106,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T01:25:18.080119+0000","last_change":"2024-04-26T01:25:18.611018+0000","last_active":"2024-04-26T01:25:18.080119+0000","last_peered":"2024-04-26T01:25:18.080119+0000","last_clean":"2024-04-26T01:25:18.080119+0000","last_became_active":"2024-04-26T01:25:15.544705+0000","last_became_peered":"2024-04-26T01:25:15.544705+0000","last_unstale":"2024-04-26T01:25:18.080119+0000","last_undegraded":"2024-04-26T01:25:18.080119+0000","last_fullsized":"2024-04-26T01:25:18.080119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.30815951600000002,"stat_sum":{"num_bytes":7602176,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":5902,"num_write":19,"num_write_kb":10704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2678784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,12,13],"acting":[4,11,9,5,10,12,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":174,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.294820+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:17.294820+0000","last_peered":"2024-04-26T01:25:17.294820+0000","last_clean":"2024-04-26T01:25:17.294820+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:17.294820+0000","last_undegraded":"2024-04-26T01:25:17.294820+0000","last_fullsized":"2024-04-26T01:25:17.294820+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"96'68","reported_seq":466,"reported_epoch":106,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T01:25:18.114500+0000","last_change":"2024-04-26T01:25:18.019185+0000","last_active":"2024-04-26T01:25:18.114500+0000","last_peered":"2024-04-26T01:25:18.114500+0000","last_clean":"2024-04-26T01:25:18.114500+0000","last_became_active":"2024-04-26T01:25:15.553849+0000","last_became_peered":"2024-04-26T01:25:15.553849+0000","last_unstale":"2024-04-26T01:25:18.114500+0000","last_undegraded":"2024-04-26T01:25:18.114500+0000","last_fullsized":"2024-04-26T01:25:18.114500+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"96'68","last_scrub_stamp":"2024-04-26T01:25:09.880728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:09.880728+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":815,"objects_trimmed":0,"snaptrim_duration":0.011072202,"stat_sum":{"num_bytes":6209536,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":605,"num_write":15,"num_write_kb":6424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":167,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.364027+0000","last_change":"2024-04-26T01:23:51.914607+0000","last_active":"2024-04-26T01:25:17.364027+0000","last_peered":"2024-04-26T01:25:17.364027+0000","last_clean":"2024-04-26T01:25:17.364027+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:17.364027+0000","last_undegraded":"2024-04-26T01:25:17.364027+0000","last_fullsized":"2024-04-26T01:25:17.364027+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:23:50.882551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":389,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.112788+0000","last_change":"2024-04-26T01:24:56.353548+0000","last_active":"2024-04-26T01:25:18.112788+0000","last_peered":"2024-04-26T01:25:18.112788+0000","last_clean":"2024-04-26T01:25:18.112788+0000","last_became_active":"2024-04-26T01:24:19.156597+0000","last_became_peered":"2024-04-26T01:24:19.156597+0000","last_unstale":"2024-04-26T01:25:18.112788+0000","last_undegraded":"2024-04-26T01:25:18.112788+0000","last_fullsized":"2024-04-26T01:25:18.112788+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:10.087788+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,3],"acting":[13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.8","version":"104'143","reported_seq":586,"reported_epoch":105,"state":"active+recovering+degraded","last_fresh":"2024-04-26T01:25:17.534733+0000","last_change":"2024-04-26T01:25:17.640744+0000","last_active":"2024-04-26T01:25:17.534733+0000","last_peered":"2024-04-26T01:25:17.534733+0000","last_clean":"2024-04-26T01:25:01.261092+0000","last_became_active":"2024-04-26T01:25:16.396635+0000","last_became_peered":"2024-04-26T01:25:16.396635+0000","last_unstale":"2024-04-26T01:25:17.534733+0000","last_undegraded":"2024-04-26T01:25:16.057678+0000","last_fullsized":"2024-04-26T01:25:17.534733+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31276866399999997,"stat_sum":{"num_bytes":11771904,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":26,"num_read_kb":13291,"num_write":58,"num_write_kb":14232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":7766016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,12,1,14,15,6],"acting":[5,0,12,1,14,15,6],"avail_no_missing":["5(0)","0(1)","1(2)","6(6)","9(3)","14(4)","15(5)"],"object_location_counts":[{"shards":"0(1),1(3),5(0),6(6),12(2),14(4),15(5)","objects":5},{"shards":"0(1),5(0),6(6),14(4),15(5)","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"91'65","reported_seq":429,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.157180+0000","last_change":"2024-04-26T01:25:15.538860+0000","last_active":"2024-04-26T01:25:18.157180+0000","last_peered":"2024-04-26T01:25:18.157180+0000","last_clean":"2024-04-26T01:25:18.157180+0000","last_became_active":"2024-04-26T01:25:15.243686+0000","last_became_peered":"2024-04-26T01:25:15.243686+0000","last_unstale":"2024-04-26T01:25:18.157180+0000","last_undegraded":"2024-04-26T01:25:18.157180+0000","last_fullsized":"2024-04-26T01:25:18.157180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.102639283,"stat_sum":{"num_bytes":2859008,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":507,"num_write":12,"num_write_kb":3832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,0,3,15,8,10],"acting":[14,1,0,3,15,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.a","version":"104'43","reported_seq":415,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.194825+0000","last_change":"2024-04-26T01:25:14.247759+0000","last_active":"2024-04-26T01:25:17.194825+0000","last_peered":"2024-04-26T01:25:17.194825+0000","last_clean":"2024-04-26T01:25:17.194825+0000","last_became_active":"2024-04-26T01:24:36.102041+0000","last_became_peered":"2024-04-26T01:24:36.102041+0000","last_unstale":"2024-04-26T01:25:17.194825+0000","last_undegraded":"2024-04-26T01:25:17.194825+0000","last_fullsized":"2024-04-26T01:25:17.194825+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'39","last_scrub_stamp":"2024-04-26T01:25:06.819166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:06.819166+0000","objects_scrubbed":1,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:07.075209+0000","scrub_duration":511,"objects_trimmed":0,"snaptrim_duration":0.001473491,"stat_sum":{"num_bytes":3293184,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":9684,"num_write":35,"num_write_kb":3408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,9,14,6,11],"acting":[3,7,1,9,14,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.b","version":"103'30","reported_seq":441,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.079964+0000","last_change":"2024-04-26T01:25:14.055394+0000","last_active":"2024-04-26T01:25:18.079964+0000","last_peered":"2024-04-26T01:25:18.079964+0000","last_clean":"2024-04-26T01:25:18.079964+0000","last_became_active":"2024-04-26T01:25:03.018610+0000","last_became_peered":"2024-04-26T01:25:03.018610+0000","last_unstale":"2024-04-26T01:25:18.079964+0000","last_undegraded":"2024-04-26T01:25:18.079964+0000","last_fullsized":"2024-04-26T01:25:18.079964+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:25:28.324726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014969370000000001,"stat_sum":{"num_bytes":1458176,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":16,"num_read_kb":4936,"num_write":31,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3407872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1,12,6,7,9],"acting":[4,0,1,12,6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.c","version":"103'107","reported_seq":512,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.072547+0000","last_change":"2024-04-26T01:25:14.252253+0000","last_active":"2024-04-26T01:25:18.072547+0000","last_peered":"2024-04-26T01:25:18.072547+0000","last_clean":"2024-04-26T01:25:18.072547+0000","last_became_active":"2024-04-26T01:25:03.109927+0000","last_became_peered":"2024-04-26T01:25:03.109927+0000","last_unstale":"2024-04-26T01:25:18.072547+0000","last_undegraded":"2024-04-26T01:25:18.072547+0000","last_fullsized":"2024-04-26T01:25:18.072547+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100830708,"stat_sum":{"num_bytes":9650176,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":3853,"num_write":83,"num_write_kb":15888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8650752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,0,10,6,12,5],"acting":[11,8,0,10,6,12,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.d","version":"103'83","reported_seq":293,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-04-26T01:25:17.165186+0000","last_change":"2024-04-26T01:25:14.054116+0000","last_active":"2024-04-26T01:25:17.165186+0000","last_peered":"2024-04-26T01:25:17.165186+0000","last_clean":"2024-04-26T01:25:17.165186+0000","last_became_active":"2024-04-26T01:24:36.105677+0000","last_became_peered":"2024-04-26T01:24:36.105677+0000","last_unstale":"2024-04-26T01:25:17.165186+0000","last_undegraded":"2024-04-26T01:25:17.165186+0000","last_fullsized":"2024-04-26T01:25:17.165186+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067180240000000004,"stat_sum":{"num_bytes":5554176,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":44,"num_read_kb":19501,"num_write":33,"num_write_kb":13936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":6406144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,14,10,0,8,4],"acting":[7,3,14,10,0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.e","version":"103'85","reported_seq":423,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.114924+0000","last_change":"2024-04-26T01:25:14.144399+0000","last_active":"2024-04-26T01:25:18.114924+0000","last_peered":"2024-04-26T01:25:18.114924+0000","last_clean":"2024-04-26T01:25:18.114924+0000","last_became_active":"2024-04-26T01:25:03.422052+0000","last_became_peered":"2024-04-26T01:25:03.422052+0000","last_unstale":"2024-04-26T01:25:18.114924+0000","last_undegraded":"2024-04-26T01:25:18.114924+0000","last_fullsized":"2024-04-26T01:25:18.114924+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012741976,"stat_sum":{"num_bytes":8896512,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":48,"num_read_kb":19154,"num_write":40,"num_write_kb":11280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":12517376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,9,1,11,8,6,12],"acting":[13,9,1,11,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.f","version":"96'10","reported_seq":386,"reported_epoch":106,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T01:25:18.110937+0000","last_change":"2024-04-26T01:25:17.400992+0000","last_active":"2024-04-26T01:25:18.110937+0000","last_peered":"2024-04-26T01:25:18.110937+0000","last_clean":"2024-04-26T01:25:18.110937+0000","last_became_active":"2024-04-26T01:24:16.442623+0000","last_became_peered":"2024-04-26T01:24:16.442623+0000","last_unstale":"2024-04-26T01:25:18.110937+0000","last_undegraded":"2024-04-26T01:25:18.110937+0000","last_fullsized":"2024-04-26T01:25:18.110937+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001419678,"stat_sum":{"num_bytes":753664,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3893,"num_write":6,"num_write_kb":736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,14,13,8,10,0],"acting":[11,1,14,13,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.10","version":"100'171","reported_seq":565,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.157321+0000","last_change":"2024-04-26T01:25:15.953125+0000","last_active":"2024-04-26T01:25:18.157321+0000","last_peered":"2024-04-26T01:25:18.157321+0000","last_clean":"2024-04-26T01:25:18.157321+0000","last_became_active":"2024-04-26T01:25:15.852541+0000","last_became_peered":"2024-04-26T01:25:15.852541+0000","last_unstale":"2024-04-26T01:25:18.157321+0000","last_undegraded":"2024-04-26T01:25:18.157321+0000","last_fullsized":"2024-04-26T01:25:18.157321+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:23:59.958369+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.097678422000000001,"stat_sum":{"num_bytes":3940352,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":27,"num_read_kb":11939,"num_write":53,"num_write_kb":7104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1875968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,15,1,5,6,4],"acting":[14,11,15,1,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.11","version":"104'67","reported_seq":493,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.079835+0000","last_change":"2024-04-26T01:25:17.780807+0000","last_active":"2024-04-26T01:25:18.079835+0000","last_peered":"2024-04-26T01:25:18.079835+0000","last_clean":"2024-04-26T01:25:18.079835+0000","last_became_active":"2024-04-26T01:25:15.851893+0000","last_became_peered":"2024-04-26T01:25:15.851893+0000","last_unstale":"2024-04-26T01:25:18.079835+0000","last_undegraded":"2024-04-26T01:25:18.079835+0000","last_fullsized":"2024-04-26T01:25:18.079835+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'67","last_scrub_stamp":"2024-04-26T01:25:17.780289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:17.780289+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:25.376755+0000","scrub_duration":1135,"objects_trimmed":0,"snaptrim_duration":0.20363114700000001,"stat_sum":{"num_bytes":8765440,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,9,5,10,12,13],"acting":[4,11,9,5,10,12,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.12","version":"96'68","reported_seq":466,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.114202+0000","last_change":"2024-04-26T01:25:16.373868+0000","last_active":"2024-04-26T01:25:18.114202+0000","last_peered":"2024-04-26T01:25:18.114202+0000","last_clean":"2024-04-26T01:25:18.114202+0000","last_became_active":"2024-04-26T01:25:15.961343+0000","last_became_peered":"2024-04-26T01:25:15.961343+0000","last_unstale":"2024-04-26T01:25:18.114202+0000","last_undegraded":"2024-04-26T01:25:18.114202+0000","last_fullsized":"2024-04-26T01:25:18.114202+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"96'68","last_scrub_stamp":"2024-04-26T01:25:09.880728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:09.880728+0000","objects_scrubbed":5,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":815,"objects_trimmed":0,"snaptrim_duration":0.20241399500000001,"stat_sum":{"num_bytes":6209536,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12750,"num_write":16,"num_write_kb":6424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,10,3,6,15,1,11],"acting":[13,10,3,6,15,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]},{"pgid":"3.13","version":"100'15","reported_seq":422,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-04-26T01:25:18.113409+0000","last_change":"2024-04-26T01:25:17.586472+0000","last_active":"2024-04-26T01:25:18.113409+0000","last_peered":"2024-04-26T01:25:18.113409+0000","last_clean":"2024-04-26T01:25:18.113409+0000","last_became_active":"2024-04-26T01:25:15.559973+0000","last_became_peered":"2024-04-26T01:25:15.559973+0000","last_unstale":"2024-04-26T01:25:18.113409+0000","last_undegraded":"2024-04-26T01:25:18.113409+0000","last_fullsized":"2024-04-26T01:25:18.113409+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":5,"last_scrub":"100'15","last_scrub_stamp":"2024-04-26T01:25:17.586037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:59.958369+0000","last_clean_scrub_stamp":"2024-04-26T01:25:17.586037+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:20.884387+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0.10391413200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,4,15,6,5,3],"acting":[13,7,4,15,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"21","length":"1"},{"start":"23","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":128892928,"num_objects":72,"num_object_clones":28,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":6,"num_read":401,"num_read_kb":148394,"num_write":625,"num_write_kb":141504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":57319424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258232320,"data_stored":338554880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1867,"ondisk_log_size":1867,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":528928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1586784,"data_compressed":99289,"data_compressed_allocated":122880,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280534,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49948,"kb_used_data":18764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321892,"statfs":{"total":96636764160,"available":96585617408,"internally_reserved":0,"allocated":19214336,"data_stored":23945462,"data_compressed":250621,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280532,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48224,"kb_used_data":17040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323616,"statfs":{"total":96636764160,"available":96587382784,"internally_reserved":0,"allocated":17448960,"data_stored":21565686,"data_compressed":250621,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280533,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51544,"kb_used_data":20368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320296,"statfs":{"total":96636764160,"available":96583983104,"internally_reserved":0,"allocated":20856832,"data_stored":27887749,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313236,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56172,"kb_used_data":24996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315668,"statfs":{"total":96636764160,"available":96579244032,"internally_reserved":0,"allocated":25595904,"data_stored":31426693,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313237,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48092,"kb_used_data":16916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323748,"statfs":{"total":96636764160,"available":96587517952,"internally_reserved":0,"allocated":17321984,"data_stored":23980165,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":90194313237,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313236,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45620,"kb_used_data":14440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326220,"statfs":{"total":96636764160,"available":96590049280,"internally_reserved":0,"allocated":14786560,"data_stored":21543045,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313237,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39732,"kb_used_data":8544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332108,"statfs":{"total":96636764160,"available":96596078592,"internally_reserved":0,"allocated":8749056,"data_stored":10396212,"data_compressed":278646,"data_compressed_allocated":376832,"data_compressed_original":1196032,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[1,2,4,6,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313236,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51168,"kb_used_data":19988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320672,"statfs":{"total":96636764160,"available":96584368128,"internally_reserved":0,"allocated":20467712,"data_stored":23343350,"data_compressed":250621,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55648,"kb_used_data":20308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316192,"statfs":{"total":96636764160,"available":96579780608,"internally_reserved":0,"allocated":20795392,"data_stored":27076741,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52216,"kb_used_data":21040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319624,"statfs":{"total":96636764160,"available":96583294976,"internally_reserved":0,"allocated":21544960,"data_stored":30562437,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313237,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":5016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":5136384,"data_stored":5666949,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53080,"kb_used_data":21900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318760,"statfs":{"total":96636764160,"available":96582410240,"internally_reserved":0,"allocated":22425600,"data_stored":26857718,"data_compressed":250621,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313236,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46336,"kb_used_data":15148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325504,"statfs":{"total":96636764160,"available":96589316096,"internally_reserved":0,"allocated":15511552,"data_stored":17684613,"data_compressed":247605,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313238,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51964,"kb_used_data":20784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319876,"statfs":{"total":96636764160,"available":96583553024,"internally_reserved":0,"allocated":21282816,"data_stored":30556406,"data_compressed":250621,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313237,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53396,"kb_used_data":18052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318444,"statfs":{"total":96636764160,"available":96582086656,"internally_reserved":0,"allocated":18485248,"data_stored":21093029,"data_compressed":280222,"data_compressed_allocated":380928,"data_compressed_original":1191936,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,4,5,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19636224,"data_stored":26402816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25726976,"data_stored":32849920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4870144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7491584,"data_stored":9781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19406848,"data_stored":22585344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19574784,"data_stored":26677248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20324352,"data_stored":30683136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3915776,"data_stored":4747264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16216064,"data_stored":20635648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":21229568,"data_stored":25964544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":14290944,"data_stored":16764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20647936,"data_stored":30392320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":17981440,"data_stored":22458368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":17203200,"data_stored":19644416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":16699392,"data_stored":23474176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13565952,"data_stored":20623360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:25:20.072 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-04-26T01:25:20.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-04-26T01:25:20.086 INFO:teuthology.orchestra.run.smithi037.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:20.086 INFO:teuthology.orchestra.run.smithi037.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:20.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-26T01:25:20.150 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:20.154+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:20.216 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:20.219+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:20.241 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:20.241 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:20.256 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-26T01:25:20.342 INFO:tasks.ceph.osd.5.smithi045.stderr:2024-04-26T01:25:20.346+0000 7f22f701e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 24937) UID: 0 2024-04-26T01:25:20.418 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-04-26T01:25:20.420+0000 7f487c68f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25314) UID: 0 2024-04-26T01:25:20.426 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:20.426 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:20.440 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-26T01:25:20.519 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:20.523+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:20.607 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:20.607 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:20.620 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-04-26T01:25:20.623+0000 7ff35abb8640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25060) UID: 0 2024-04-26T01:25:20.623 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-26T01:25:20.720 INFO:tasks.ceph.osd.12.smithi037.stderr:2024-04-26T01:25:20.723+0000 7f23aee2b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 25065) UID: 0 2024-04-26T01:25:20.751 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:20.751 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:20.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-26T01:25:20.822 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-04-26T01:25:20.826+0000 7f4a50c8f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25064) UID: 0 2024-04-26T01:25:20.892 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:20.893 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:20.908 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-26T01:25:20.954 INFO:tasks.ceph.osd.9.smithi045.stderr:2024-04-26T01:25:20.958+0000 7faa5fad0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 24939) UID: 0 2024-04-26T01:25:21.023 INFO:tasks.ceph.osd.10.smithi071.stderr:2024-04-26T01:25:21.022+0000 7efdd51d4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 25068) UID: 0 2024-04-26T01:25:21.029 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:21.030 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:21.044 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-04-26T01:25:21.084 INFO:teuthology.orchestra.run.smithi037.stderr:pool 'unique_pool_0' removed 2024-04-26T01:25:21.112 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-26T01:25:21.124 INFO:tasks.thrashosds:joining thrashosds 2024-04-26T01:25:21.124 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-26T01:25:21.220 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:21.220 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:21.236 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-04-26T01:25:21.371 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:21.371 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:21.386 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-04-26T01:25:21.503 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:21.503 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:21.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-04-26T01:25:21.645 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:21.645 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:21.659 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-04-26T01:25:21.788 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:21.788 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:21.803 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-04-26T01:25:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:21.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-26T01:25:22.074 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:22.074 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:22.089 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-26T01:25:22.093 INFO:teuthology.orchestra.run.smithi037.stderr:noscrub is unset 2024-04-26T01:25:22.117 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-26T01:25:22.207 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:22.207 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:22.221 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-26T01:25:22.346 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:22.346 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:22.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-26T01:25:22.486 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:22.487 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:22.500 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-26T01:25:22.622 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:22.622 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:22.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-26T01:25:22.758 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:22.758 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:22.772 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-04-26T01:25:22.894 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:22.894 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:22.908 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-04-26T01:25:23.026 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.026 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.041 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-04-26T01:25:23.096 INFO:teuthology.orchestra.run.smithi037.stderr:nodeep-scrub is unset 2024-04-26T01:25:23.159 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.160 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-04-26T01:25:23.294 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.294 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.308 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-04-26T01:25:23.427 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.427 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-04-26T01:25:23.556 INFO:teuthology.orchestra.run.smithi071.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.557 INFO:teuthology.orchestra.run.smithi071.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-26T01:25:23.704 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.704 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-26T01:25:23.837 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.837 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-26T01:25:23.976 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:23.976 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:23.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-26T01:25:24.112 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-26T01:25:24.112 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-04-26T01:25:24.112 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-04-26T01:25:24.114 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:24.114 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:24.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-26T01:25:24.252 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:24.252 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:24.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-26T01:25:24.396 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:24.396 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:24.410 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-04-26T01:25:24.434 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-04-26T01:25:24.439+0000 7f533fe62640 -1 mon.a@0(leader).osd e111 definitely_dead 0 2024-04-26T01:25:24.532 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:24.533 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:24.547 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-04-26T01:25:24.664 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:24.664 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:24.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-04-26T01:25:24.795 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:24.796 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:24.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-04-26T01:25:24.933 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:24.933 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:24.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-04-26T01:25:25.068 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:25.068 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:25.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-04-26T01:25:25.113 INFO:teuthology.orchestra.run.smithi037.stderr:marked in osd.2. 2024-04-26T01:25:25.140 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-04-26T01:25:25.140 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-26T01:25:25.199 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-04-26T01:25:25.199 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T01:25:25.286 INFO:teuthology.orchestra.run.smithi037.stdout:mon.a: {} 2024-04-26T01:25:25.286 INFO:teuthology.orchestra.run.smithi037.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:25:25.299 INFO:teuthology.orchestra.run.smithi037.stdout:mon.b: {} 2024-04-26T01:25:25.299 INFO:teuthology.orchestra.run.smithi037.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:25:25.314 INFO:teuthology.orchestra.run.smithi037.stdout:mon.c: {} 2024-04-26T01:25:25.314 INFO:teuthology.orchestra.run.smithi037.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T01:25:25.331 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-26T01:25:26.121 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T01:25:26.147 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-26T01:25:27.133 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-26T01:25:27.157 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-26T01:25:28.136 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T01:25:28.162 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-26T01:25:28.787 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-26T01:25:28.816 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-04-26T01:25:29.790 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.8 to 1 (10000) 2024-04-26T01:25:29.814 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-04-26T01:25:30.792 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-04-26T01:25:30.819 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-04-26T01:25:31.803 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.12 to 1 (10000) 2024-04-26T01:25:31.831 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-04-26T01:25:32.822 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-04-26T01:25:32.848 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-26T01:25:33.834 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T01:25:33.854 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-26T01:25:34.839 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-26T01:25:34.863 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-26T01:25:35.842 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.5 to 1 (10000) 2024-04-26T01:25:35.866 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-26T01:25:36.851 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-26T01:25:36.878 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-04-26T01:25:37.860 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.9 to 1 (10000) 2024-04-26T01:25:37.885 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-04-26T01:25:38.870 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-04-26T01:25:38.894 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-04-26T01:25:39.880 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.13 to 1 (10000) 2024-04-26T01:25:39.950 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-04-26T01:25:40.891 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-04-26T01:25:40.915 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-26T01:25:41.902 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.2 to 1 (10000) 2024-04-26T01:25:41.927 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-26T01:25:42.913 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-26T01:25:42.937 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-26T01:25:43.931 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T01:25:43.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-26T01:25:44.934 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-26T01:25:44.958 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-04-26T01:25:45.943 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.10 to 1 (10000) 2024-04-26T01:25:45.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-04-26T01:25:46.963 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-04-26T01:25:46.989 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-04-26T01:25:47.966 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.14 to 1 (10000) 2024-04-26T01:25:47.990 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-04-26T01:25:48.976 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-04-26T01:25:49.001 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-26T01:25:49.986 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T01:25:50.011 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-26T01:25:50.996 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-26T01:25:51.024 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-26T01:25:52.011 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.7 to 1 (10000) 2024-04-26T01:25:52.034 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-26T01:25:53.020 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-26T01:25:53.043 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-04-26T01:25:54.028 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.11 to 1 (10000) 2024-04-26T01:25:54.052 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-04-26T01:25:55.031 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-04-26T01:25:55.061 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-04-26T01:25:56.048 INFO:teuthology.orchestra.run.smithi037.stderr:reweighted osd.15 to 1 (10000) 2024-04-26T01:25:56.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-04-26T01:25:57.056 INFO:teuthology.orchestra.run.smithi037.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-04-26T01:25:57.083 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-26T01:25:57.083 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-26T01:25:57.083 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-26T01:25:57.083 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-26T01:25:57.083 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T01:25:57.083 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:25:57.394 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:57.395 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":144,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:25:57.054171+0000","last_up_change":"2024-04-26T01:23:47.867650+0000","last_in_change":"2024-04-26T01:25:25.109979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:23:47.920762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:23:50.025173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6819","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6820","nonce":906916438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6824","nonce":906916438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6829","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6831","nonce":906916438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6827","nonce":906916438}]},"public_addr":"172.21.15.37:6820/906916438","cluster_addr":"172.21.15.37:6824/906916438","heartbeat_back_addr":"172.21.15.37:6831/906916438","heartbeat_front_addr":"172.21.15.37:6827/906916438","state":["exists","up"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6813","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6816","nonce":1630954663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6823","nonce":1630954663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6831","nonce":1630954663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6828","nonce":1630954663}]},"public_addr":"172.21.15.45:6816/1630954663","cluster_addr":"172.21.15.45:6823/1630954663","heartbeat_back_addr":"172.21.15.45:6831/1630954663","heartbeat_front_addr":"172.21.15.45:6828/1630954663","state":["exists","up"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6801","nonce":2961312835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6803","nonce":2961312835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6807","nonce":2961312835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6805","nonce":2961312835}]},"public_addr":"172.21.15.71:6801/2961312835","cluster_addr":"172.21.15.71:6803/2961312835","heartbeat_back_addr":"172.21.15.71:6807/2961312835","heartbeat_front_addr":"172.21.15.71:6805/2961312835","state":["exists","up"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6801","nonce":1026435117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6806","nonce":1026435117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6821","nonce":1026435117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6814","nonce":1026435117}]},"public_addr":"172.21.15.181:6801/1026435117","cluster_addr":"172.21.15.181:6806/1026435117","heartbeat_back_addr":"172.21.15.181:6821/1026435117","heartbeat_front_addr":"172.21.15.181:6814/1026435117","state":["exists","up"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6803","nonce":1575437742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6805","nonce":1575437742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6809","nonce":1575437742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6807","nonce":1575437742}]},"public_addr":"172.21.15.37:6803/1575437742","cluster_addr":"172.21.15.37:6805/1575437742","heartbeat_back_addr":"172.21.15.37:6809/1575437742","heartbeat_front_addr":"172.21.15.37:6807/1575437742","state":["exists","up"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6801","nonce":3759230996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6803","nonce":3759230996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6807","nonce":3759230996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6805","nonce":3759230996}]},"public_addr":"172.21.15.45:6801/3759230996","cluster_addr":"172.21.15.45:6803/3759230996","heartbeat_back_addr":"172.21.15.45:6807/3759230996","heartbeat_front_addr":"172.21.15.45:6805/3759230996","state":["exists","up"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6809","nonce":76180415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6811","nonce":76180415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6815","nonce":76180415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6813","nonce":76180415}]},"public_addr":"172.21.15.71:6809/76180415","cluster_addr":"172.21.15.71:6811/76180415","heartbeat_back_addr":"172.21.15.71:6815/76180415","heartbeat_front_addr":"172.21.15.71:6813/76180415","state":["exists","up"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6807","nonce":4043929361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6812","nonce":4043929361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6823","nonce":4043929361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6820","nonce":4043929361}]},"public_addr":"172.21.15.181:6807/4043929361","cluster_addr":"172.21.15.181:6812/4043929361","heartbeat_back_addr":"172.21.15.181:6823/4043929361","heartbeat_front_addr":"172.21.15.181:6820/4043929361","state":["exists","up"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6811","nonce":4094482142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6813","nonce":4094482142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6817","nonce":4094482142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6815","nonce":4094482142}]},"public_addr":"172.21.15.37:6811/4094482142","cluster_addr":"172.21.15.37:6813/4094482142","heartbeat_back_addr":"172.21.15.37:6817/4094482142","heartbeat_front_addr":"172.21.15.37:6815/4094482142","state":["exists","up"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6810","nonce":3448396888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6815","nonce":3448396888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6824","nonce":3448396888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6817","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6819","nonce":3448396888}]},"public_addr":"172.21.15.45:6810/3448396888","cluster_addr":"172.21.15.45:6815/3448396888","heartbeat_back_addr":"172.21.15.45:6824/3448396888","heartbeat_front_addr":"172.21.15.45:6819/3448396888","state":["exists","up"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6817","nonce":896289472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6819","nonce":896289472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6823","nonce":896289472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6821","nonce":896289472}]},"public_addr":"172.21.15.71:6817/896289472","cluster_addr":"172.21.15.71:6819/896289472","heartbeat_back_addr":"172.21.15.71:6823/896289472","heartbeat_front_addr":"172.21.15.71:6821/896289472","state":["exists","up"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6825","nonce":647568313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6827","nonce":647568313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6831","nonce":647568313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6829","nonce":647568313}]},"public_addr":"172.21.15.181:6825/647568313","cluster_addr":"172.21.15.181:6827/647568313","heartbeat_back_addr":"172.21.15.181:6831/647568313","heartbeat_front_addr":"172.21.15.181:6829/647568313","state":["exists","up"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6821","nonce":3336489453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6825","nonce":3336489453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6833","nonce":3336489453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6830","nonce":3336489453}]},"public_addr":"172.21.15.37:6821/3336489453","cluster_addr":"172.21.15.37:6825/3336489453","heartbeat_back_addr":"172.21.15.37:6833/3336489453","heartbeat_front_addr":"172.21.15.37:6830/3336489453","state":["exists","up"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6811","nonce":1608953722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6818","nonce":1608953722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6827","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6829","nonce":1608953722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6821","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6825","nonce":1608953722}]},"public_addr":"172.21.15.45:6811/1608953722","cluster_addr":"172.21.15.45:6818/1608953722","heartbeat_back_addr":"172.21.15.45:6829/1608953722","heartbeat_front_addr":"172.21.15.45:6825/1608953722","state":["exists","up"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6825","nonce":3719375992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6827","nonce":3719375992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6831","nonce":3719375992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6829","nonce":3719375992}]},"public_addr":"172.21.15.71:6825/3719375992","cluster_addr":"172.21.15.71:6827/3719375992","heartbeat_back_addr":"172.21.15.71:6831/3719375992","heartbeat_front_addr":"172.21.15.71:6829/3719375992","state":["exists","up"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6805","nonce":2891128021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6811","nonce":2891128021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6817","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6819","nonce":2891128021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6815","nonce":2891128021}]},"public_addr":"172.21.15.181:6805/2891128021","cluster_addr":"172.21.15.181:6811/2891128021","heartbeat_back_addr":"172.21.15.181:6819/2891128021","heartbeat_front_addr":"172.21.15.181:6815/2891128021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.150989+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:50.485009+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:55.015898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.670769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.579754+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:08.910977+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:51.291925+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:51.369682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.515953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:14.187344+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:13.456713+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:02.015135+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.667171+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.495812+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:48.329599+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:47.498480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":25,"length":1},{"begin":27,"length":2},{"begin":33,"length":1},{"begin":35,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:25:57.413 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T01:25:57.413 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:25:57.721 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:25:57.722 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":144,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:25:57.054171+0000","last_up_change":"2024-04-26T01:23:47.867650+0000","last_in_change":"2024-04-26T01:25:25.109979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:23:47.920762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:23:50.025173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6819","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6820","nonce":906916438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6824","nonce":906916438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6829","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6831","nonce":906916438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6827","nonce":906916438}]},"public_addr":"172.21.15.37:6820/906916438","cluster_addr":"172.21.15.37:6824/906916438","heartbeat_back_addr":"172.21.15.37:6831/906916438","heartbeat_front_addr":"172.21.15.37:6827/906916438","state":["exists","up"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6813","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6816","nonce":1630954663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6823","nonce":1630954663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6831","nonce":1630954663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6828","nonce":1630954663}]},"public_addr":"172.21.15.45:6816/1630954663","cluster_addr":"172.21.15.45:6823/1630954663","heartbeat_back_addr":"172.21.15.45:6831/1630954663","heartbeat_front_addr":"172.21.15.45:6828/1630954663","state":["exists","up"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6801","nonce":2961312835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6803","nonce":2961312835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6807","nonce":2961312835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6805","nonce":2961312835}]},"public_addr":"172.21.15.71:6801/2961312835","cluster_addr":"172.21.15.71:6803/2961312835","heartbeat_back_addr":"172.21.15.71:6807/2961312835","heartbeat_front_addr":"172.21.15.71:6805/2961312835","state":["exists","up"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6801","nonce":1026435117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6806","nonce":1026435117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6821","nonce":1026435117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6814","nonce":1026435117}]},"public_addr":"172.21.15.181:6801/1026435117","cluster_addr":"172.21.15.181:6806/1026435117","heartbeat_back_addr":"172.21.15.181:6821/1026435117","heartbeat_front_addr":"172.21.15.181:6814/1026435117","state":["exists","up"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6803","nonce":1575437742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6805","nonce":1575437742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6809","nonce":1575437742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6807","nonce":1575437742}]},"public_addr":"172.21.15.37:6803/1575437742","cluster_addr":"172.21.15.37:6805/1575437742","heartbeat_back_addr":"172.21.15.37:6809/1575437742","heartbeat_front_addr":"172.21.15.37:6807/1575437742","state":["exists","up"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6801","nonce":3759230996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6803","nonce":3759230996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6807","nonce":3759230996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6805","nonce":3759230996}]},"public_addr":"172.21.15.45:6801/3759230996","cluster_addr":"172.21.15.45:6803/3759230996","heartbeat_back_addr":"172.21.15.45:6807/3759230996","heartbeat_front_addr":"172.21.15.45:6805/3759230996","state":["exists","up"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6809","nonce":76180415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6811","nonce":76180415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6815","nonce":76180415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6813","nonce":76180415}]},"public_addr":"172.21.15.71:6809/76180415","cluster_addr":"172.21.15.71:6811/76180415","heartbeat_back_addr":"172.21.15.71:6815/76180415","heartbeat_front_addr":"172.21.15.71:6813/76180415","state":["exists","up"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6807","nonce":4043929361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6812","nonce":4043929361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6823","nonce":4043929361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6820","nonce":4043929361}]},"public_addr":"172.21.15.181:6807/4043929361","cluster_addr":"172.21.15.181:6812/4043929361","heartbeat_back_addr":"172.21.15.181:6823/4043929361","heartbeat_front_addr":"172.21.15.181:6820/4043929361","state":["exists","up"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6811","nonce":4094482142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6813","nonce":4094482142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6817","nonce":4094482142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6815","nonce":4094482142}]},"public_addr":"172.21.15.37:6811/4094482142","cluster_addr":"172.21.15.37:6813/4094482142","heartbeat_back_addr":"172.21.15.37:6817/4094482142","heartbeat_front_addr":"172.21.15.37:6815/4094482142","state":["exists","up"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6810","nonce":3448396888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6815","nonce":3448396888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6824","nonce":3448396888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6817","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6819","nonce":3448396888}]},"public_addr":"172.21.15.45:6810/3448396888","cluster_addr":"172.21.15.45:6815/3448396888","heartbeat_back_addr":"172.21.15.45:6824/3448396888","heartbeat_front_addr":"172.21.15.45:6819/3448396888","state":["exists","up"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6817","nonce":896289472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6819","nonce":896289472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6823","nonce":896289472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6821","nonce":896289472}]},"public_addr":"172.21.15.71:6817/896289472","cluster_addr":"172.21.15.71:6819/896289472","heartbeat_back_addr":"172.21.15.71:6823/896289472","heartbeat_front_addr":"172.21.15.71:6821/896289472","state":["exists","up"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6825","nonce":647568313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6827","nonce":647568313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6831","nonce":647568313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6829","nonce":647568313}]},"public_addr":"172.21.15.181:6825/647568313","cluster_addr":"172.21.15.181:6827/647568313","heartbeat_back_addr":"172.21.15.181:6831/647568313","heartbeat_front_addr":"172.21.15.181:6829/647568313","state":["exists","up"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6821","nonce":3336489453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6825","nonce":3336489453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6833","nonce":3336489453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6830","nonce":3336489453}]},"public_addr":"172.21.15.37:6821/3336489453","cluster_addr":"172.21.15.37:6825/3336489453","heartbeat_back_addr":"172.21.15.37:6833/3336489453","heartbeat_front_addr":"172.21.15.37:6830/3336489453","state":["exists","up"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6811","nonce":1608953722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6818","nonce":1608953722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6827","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6829","nonce":1608953722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6821","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6825","nonce":1608953722}]},"public_addr":"172.21.15.45:6811/1608953722","cluster_addr":"172.21.15.45:6818/1608953722","heartbeat_back_addr":"172.21.15.45:6829/1608953722","heartbeat_front_addr":"172.21.15.45:6825/1608953722","state":["exists","up"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6825","nonce":3719375992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6827","nonce":3719375992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6831","nonce":3719375992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6829","nonce":3719375992}]},"public_addr":"172.21.15.71:6825/3719375992","cluster_addr":"172.21.15.71:6827/3719375992","heartbeat_back_addr":"172.21.15.71:6831/3719375992","heartbeat_front_addr":"172.21.15.71:6829/3719375992","state":["exists","up"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6805","nonce":2891128021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6811","nonce":2891128021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6817","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6819","nonce":2891128021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6815","nonce":2891128021}]},"public_addr":"172.21.15.181:6805/2891128021","cluster_addr":"172.21.15.181:6811/2891128021","heartbeat_back_addr":"172.21.15.181:6819/2891128021","heartbeat_front_addr":"172.21.15.181:6815/2891128021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.150989+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:50.485009+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:55.015898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.670769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.579754+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:08.910977+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:51.291925+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:51.369682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.515953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:14.187344+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:13.456713+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:02.015135+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.667171+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.495812+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:48.329599+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:47.498480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":25,"length":1},{"begin":27,"length":2},{"begin":33,"length":1},{"begin":35,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:25:57.739 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T01:25:57.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T01:25:57.740 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T01:25:57.741 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T01:25:57.741 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T01:25:57.741 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T01:25:57.742 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-26T01:25:57.742 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-26T01:25:57.743 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-04-26T01:25:57.743 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-04-26T01:25:57.743 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-04-26T01:25:57.744 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-04-26T01:25:57.744 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-04-26T01:25:57.745 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-04-26T01:25:57.745 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-04-26T01:25:57.745 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-04-26T01:25:58.146 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:25:58.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T01:25:58.175 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-04-26T01:25:58.176 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T01:25:58.201 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:25:58.201 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T01:25:58.212 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:58.212 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-26T01:25:58.215 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:25:58.215 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T01:25:58.227 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:58.227 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-26T01:25:58.247 INFO:teuthology.orchestra.run.smithi037.stdout:94489280542 2024-04-26T01:25:58.247 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-26T01:25:58.252 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:25:58.253 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-26T01:25:58.255 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:25:58.256 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T01:25:58.261 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:25:58.261 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T01:25:58.282 INFO:teuthology.orchestra.run.smithi037.stdout:94489280540 2024-04-26T01:25:58.282 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-26T01:25:58.337 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:58.338 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T01:25:58.367 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:58.367 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-26T01:25:58.368 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:58.368 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T01:25:58.385 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:58.385 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-26T01:25:58.392 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:58.392 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-26T01:25:59.115 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:59.149 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:59.172 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313244 for osd.2 2024-04-26T01:25:59.175 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:59.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313244 for osd.6 2024-04-26T01:25:59.199 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:59.215 INFO:teuthology.orchestra.run.smithi037.stdout:90194313243 2024-04-26T01:25:59.216 INFO:teuthology.orchestra.run.smithi037.stdout:94489280540 2024-04-26T01:25:59.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313245 for osd.11 2024-04-26T01:25:59.245 INFO:teuthology.orchestra.run.smithi037.stdout:90194313243 2024-04-26T01:25:59.270 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313245 for osd.3 2024-04-26T01:25:59.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280540 for osd.0 2024-04-26T01:25:59.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313243 for osd.10 2024-04-26T01:25:59.297 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313243 for osd.15 2024-04-26T01:25:59.305 INFO:teuthology.orchestra.run.smithi037.stdout:90194313243 2024-04-26T01:25:59.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313243 for osd.4 2024-04-26T01:25:59.347 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:59.350 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:59.353 INFO:teuthology.orchestra.run.smithi037.stdout:90194313243 2024-04-26T01:25:59.379 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313244 for osd.9 2024-04-26T01:25:59.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313244 for osd.13 2024-04-26T01:25:59.387 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313243 for osd.1 2024-04-26T01:25:59.387 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-04-26T01:25:59.388 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:59.394 INFO:teuthology.orchestra.run.smithi037.stdout:90194313244 2024-04-26T01:25:59.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280541 for osd.12 2024-04-26T01:25:59.412 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313244 for osd.14 2024-04-26T01:25:59.418 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313244 for osd.7 2024-04-26T01:25:59.437 INFO:teuthology.orchestra.run.smithi037.stdout:94489280539 2024-04-26T01:25:59.445 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:25:59.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280539 for osd.8 2024-04-26T01:25:59.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313245 for osd.5 2024-04-26T01:26:00.174 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T01:26:00.199 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-26T01:26:00.237 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-04-26T01:26:00.271 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T01:26:00.277 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T01:26:00.279 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-04-26T01:26:00.298 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-04-26T01:26:00.344 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T01:26:00.380 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-04-26T01:26:00.385 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-04-26T01:26:00.387 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T01:26:00.412 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-04-26T01:26:00.412 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-04-26T01:26:00.419 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-26T01:26:00.456 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-04-26T01:26:00.464 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T01:26:00.906 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:00.941 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:00.968 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313246 for osd.2 2024-04-26T01:26:00.968 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:00.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313246 for osd.6 2024-04-26T01:26:00.988 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.053 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:01.062 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:26:01.117 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:26:01.120 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313246 for osd.3 2024-04-26T01:26:01.120 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313245 for osd.15 2024-04-26T01:26:01.149 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.172 INFO:teuthology.orchestra.run.smithi037.stdout:94489280542 2024-04-26T01:26:01.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313245 for osd.10 2024-04-26T01:26:01.198 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280541 got 94489280542 for osd.0 2024-04-26T01:26:01.215 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.222 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:01.255 INFO:teuthology.orchestra.run.smithi037.stdout:90194313247 2024-04-26T01:26:01.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313246 for osd.9 2024-04-26T01:26:01.265 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.281 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:26:01.281 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:01.294 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313247 for osd.11 2024-04-26T01:26:01.295 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.309 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313246 for osd.7 2024-04-26T01:26:01.309 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.310 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:01.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313245 for osd.4 2024-04-26T01:26:01.321 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313246 for osd.14 2024-04-26T01:26:01.329 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.348 INFO:teuthology.orchestra.run.smithi037.stdout:90194313245 2024-04-26T01:26:01.356 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:01.368 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313244 got 90194313245 for osd.1 2024-04-26T01:26:01.368 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.376 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313245 got 90194313246 for osd.13 2024-04-26T01:26:01.376 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.408 INFO:teuthology.orchestra.run.smithi037.stdout:90194313246 2024-04-26T01:26:01.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313246 got 90194313246 for osd.5 2024-04-26T01:26:01.425 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.440 INFO:teuthology.orchestra.run.smithi037.stdout:94489280541 2024-04-26T01:26:01.446 INFO:teuthology.orchestra.run.smithi037.stdout:94489280543 2024-04-26T01:26:01.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280540 got 94489280541 for osd.8 2024-04-26T01:26:01.460 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280542 got 94489280543 for osd.12 2024-04-26T01:26:01.465 DEBUG:teuthology.parallel:result is None 2024-04-26T01:26:01.465 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T01:26:01.465 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:26:01.776 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:01.777 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:26:01.794 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-04-26T01:25:59.855347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529812,"kb_used_data":26732,"kb_used_omap":24,"kb_used_meta":502823,"kb_avail":1509419628,"statfs":{"total":1546188226560,"available":1545645699072,"internally_reserved":0,"allocated":27373568,"data_stored":21835584,"data_compressed":5882800,"data_compressed_allocated":8077312,"data_compressed_original":17027072,"omap_allocated":25521,"internal_metadata":514890831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.820002"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":247,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.207184+0000","last_change":"2024-04-26T01:25:21.992991+0000","last_active":"2024-04-26T01:25:57.207184+0000","last_peered":"2024-04-26T01:25:57.207184+0000","last_clean":"2024-04-26T01:25:57.207184+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:57.207184+0000","last_undegraded":"2024-04-26T01:25:57.207184+0000","last_fullsized":"2024-04-26T01:25:57.207184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:21.992665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:21.992665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:34.893460+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":249,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206048+0000","last_change":"2024-04-26T01:25:19.031279+0000","last_active":"2024-04-26T01:25:57.206048+0000","last_peered":"2024-04-26T01:25:57.206048+0000","last_clean":"2024-04-26T01:25:57.206048+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:57.206048+0000","last_undegraded":"2024-04-26T01:25:57.206048+0000","last_fullsized":"2024-04-26T01:25:57.206048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:19.031090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:19.031090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.870416+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.507326+0000","last_change":"2024-04-26T01:25:22.810840+0000","last_active":"2024-04-26T01:25:57.507326+0000","last_peered":"2024-04-26T01:25:57.507326+0000","last_clean":"2024-04-26T01:25:57.507326+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:57.507326+0000","last_undegraded":"2024-04-26T01:25:57.507326+0000","last_fullsized":"2024-04-26T01:25:57.507326+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.810688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.810688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.060679+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066054+0000","last_change":"2024-04-26T01:25:10.080365+0000","last_active":"2024-04-26T01:25:57.066054+0000","last_peered":"2024-04-26T01:25:57.066054+0000","last_clean":"2024-04-26T01:25:57.066054+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:57.066054+0000","last_undegraded":"2024-04-26T01:25:57.066054+0000","last_fullsized":"2024-04-26T01:25:57.066054+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:10.080096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:10.080096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.198292+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":261,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206283+0000","last_change":"2024-04-26T01:25:20.498495+0000","last_active":"2024-04-26T01:25:57.206283+0000","last_peered":"2024-04-26T01:25:57.206283+0000","last_clean":"2024-04-26T01:25:57.206283+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:57.206283+0000","last_undegraded":"2024-04-26T01:25:57.206283+0000","last_fullsized":"2024-04-26T01:25:57.206283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:20.498167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:20.498167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:46.374939+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":251,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206973+0000","last_change":"2024-04-26T01:25:23.320622+0000","last_active":"2024-04-26T01:25:57.206973+0000","last_peered":"2024-04-26T01:25:57.206973+0000","last_clean":"2024-04-26T01:25:57.206973+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:57.206973+0000","last_undegraded":"2024-04-26T01:25:57.206973+0000","last_fullsized":"2024-04-26T01:25:57.206973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:23.320400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:23.320400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:26.196523+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":248,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206154+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:57.206154+0000","last_peered":"2024-04-26T01:25:57.206154+0000","last_clean":"2024-04-26T01:25:57.206154+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:57.206154+0000","last_undegraded":"2024-04-26T01:25:57.206154+0000","last_fullsized":"2024-04-26T01:25:57.206154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066020+0000","last_change":"2024-04-26T01:25:22.598146+0000","last_active":"2024-04-26T01:25:57.066020+0000","last_peered":"2024-04-26T01:25:57.066020+0000","last_clean":"2024-04-26T01:25:57.066020+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:57.066020+0000","last_undegraded":"2024-04-26T01:25:57.066020+0000","last_fullsized":"2024-04-26T01:25:57.066020+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.597918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.597918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:36.990896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":476,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206856+0000","last_change":"2024-04-26T01:25:29.907934+0000","last_active":"2024-04-26T01:25:57.206856+0000","last_peered":"2024-04-26T01:25:57.206856+0000","last_clean":"2024-04-26T01:25:57.206856+0000","last_became_active":"2024-04-26T01:25:29.907215+0000","last_became_peered":"2024-04-26T01:25:29.907215+0000","last_unstale":"2024-04-26T01:25:57.206856+0000","last_undegraded":"2024-04-26T01:25:57.206856+0000","last_fullsized":"2024-04-26T01:25:57.206856+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.311527+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":100727,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334836,"statfs":{"total":96636764160,"available":96598872064,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28740,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343100,"statfs":{"total":96636764160,"available":96607334400,"internally_reserved":0,"allocated":1753088,"data_stored":1827536,"data_compressed":397563,"data_compressed_allocated":540672,"data_compressed_original":1523712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28684,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343156,"statfs":{"total":96636764160,"available":96607391744,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1765376,"data_stored":1827536,"data_compressed":396125,"data_compressed_allocated":540672,"data_compressed_original":1511424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:26:01.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:26:02.074 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:02.074 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:26:02.091 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-26T01:26:01.855947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529812,"kb_used_data":26732,"kb_used_omap":24,"kb_used_meta":502823,"kb_avail":1509419628,"statfs":{"total":1546188226560,"available":1545645699072,"internally_reserved":0,"allocated":27373568,"data_stored":21835584,"data_compressed":5882800,"data_compressed_allocated":8077312,"data_compressed_original":17027072,"omap_allocated":25521,"internal_metadata":514890831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.817695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":247,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.207184+0000","last_change":"2024-04-26T01:25:21.992991+0000","last_active":"2024-04-26T01:25:57.207184+0000","last_peered":"2024-04-26T01:25:57.207184+0000","last_clean":"2024-04-26T01:25:57.207184+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:57.207184+0000","last_undegraded":"2024-04-26T01:25:57.207184+0000","last_fullsized":"2024-04-26T01:25:57.207184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:21.992665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:21.992665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:34.893460+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":249,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206048+0000","last_change":"2024-04-26T01:25:19.031279+0000","last_active":"2024-04-26T01:25:57.206048+0000","last_peered":"2024-04-26T01:25:57.206048+0000","last_clean":"2024-04-26T01:25:57.206048+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:57.206048+0000","last_undegraded":"2024-04-26T01:25:57.206048+0000","last_fullsized":"2024-04-26T01:25:57.206048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:19.031090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:19.031090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.870416+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.507326+0000","last_change":"2024-04-26T01:25:22.810840+0000","last_active":"2024-04-26T01:25:57.507326+0000","last_peered":"2024-04-26T01:25:57.507326+0000","last_clean":"2024-04-26T01:25:57.507326+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:57.507326+0000","last_undegraded":"2024-04-26T01:25:57.507326+0000","last_fullsized":"2024-04-26T01:25:57.507326+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.810688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.810688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.060679+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066054+0000","last_change":"2024-04-26T01:25:10.080365+0000","last_active":"2024-04-26T01:25:57.066054+0000","last_peered":"2024-04-26T01:25:57.066054+0000","last_clean":"2024-04-26T01:25:57.066054+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:57.066054+0000","last_undegraded":"2024-04-26T01:25:57.066054+0000","last_fullsized":"2024-04-26T01:25:57.066054+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:10.080096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:10.080096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.198292+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":261,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206283+0000","last_change":"2024-04-26T01:25:20.498495+0000","last_active":"2024-04-26T01:25:57.206283+0000","last_peered":"2024-04-26T01:25:57.206283+0000","last_clean":"2024-04-26T01:25:57.206283+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:57.206283+0000","last_undegraded":"2024-04-26T01:25:57.206283+0000","last_fullsized":"2024-04-26T01:25:57.206283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:20.498167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:20.498167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:46.374939+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":251,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206973+0000","last_change":"2024-04-26T01:25:23.320622+0000","last_active":"2024-04-26T01:25:57.206973+0000","last_peered":"2024-04-26T01:25:57.206973+0000","last_clean":"2024-04-26T01:25:57.206973+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:57.206973+0000","last_undegraded":"2024-04-26T01:25:57.206973+0000","last_fullsized":"2024-04-26T01:25:57.206973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:23.320400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:23.320400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:26.196523+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":248,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206154+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:57.206154+0000","last_peered":"2024-04-26T01:25:57.206154+0000","last_clean":"2024-04-26T01:25:57.206154+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:57.206154+0000","last_undegraded":"2024-04-26T01:25:57.206154+0000","last_fullsized":"2024-04-26T01:25:57.206154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066020+0000","last_change":"2024-04-26T01:25:22.598146+0000","last_active":"2024-04-26T01:25:57.066020+0000","last_peered":"2024-04-26T01:25:57.066020+0000","last_clean":"2024-04-26T01:25:57.066020+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:57.066020+0000","last_undegraded":"2024-04-26T01:25:57.066020+0000","last_fullsized":"2024-04-26T01:25:57.066020+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.597918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.597918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:36.990896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":476,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206856+0000","last_change":"2024-04-26T01:25:29.907934+0000","last_active":"2024-04-26T01:25:57.206856+0000","last_peered":"2024-04-26T01:25:57.206856+0000","last_clean":"2024-04-26T01:25:57.206856+0000","last_became_active":"2024-04-26T01:25:29.907215+0000","last_became_peered":"2024-04-26T01:25:29.907215+0000","last_unstale":"2024-04-26T01:25:57.206856+0000","last_undegraded":"2024-04-26T01:25:57.206856+0000","last_fullsized":"2024-04-26T01:25:57.206856+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.311527+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":100727,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334836,"statfs":{"total":96636764160,"available":96598872064,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28740,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343100,"statfs":{"total":96636764160,"available":96607334400,"internally_reserved":0,"allocated":1753088,"data_stored":1827536,"data_compressed":397563,"data_compressed_allocated":540672,"data_compressed_original":1523712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28684,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343156,"statfs":{"total":96636764160,"available":96607391744,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1765376,"data_stored":1827536,"data_compressed":396125,"data_compressed_allocated":540672,"data_compressed_original":1511424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:26:02.092 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-26T01:26:02.092 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T01:26:02.102 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T01:26:02.102 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:26:02.390 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:02.391 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:26:02.407 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-26T01:26:01.855947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529812,"kb_used_data":26732,"kb_used_omap":24,"kb_used_meta":502823,"kb_avail":1509419628,"statfs":{"total":1546188226560,"available":1545645699072,"internally_reserved":0,"allocated":27373568,"data_stored":21835584,"data_compressed":5882800,"data_compressed_allocated":8077312,"data_compressed_original":17027072,"omap_allocated":25521,"internal_metadata":514890831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.817695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":247,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.207184+0000","last_change":"2024-04-26T01:25:21.992991+0000","last_active":"2024-04-26T01:25:57.207184+0000","last_peered":"2024-04-26T01:25:57.207184+0000","last_clean":"2024-04-26T01:25:57.207184+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:57.207184+0000","last_undegraded":"2024-04-26T01:25:57.207184+0000","last_fullsized":"2024-04-26T01:25:57.207184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:21.992665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:21.992665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:34.893460+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":249,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206048+0000","last_change":"2024-04-26T01:25:19.031279+0000","last_active":"2024-04-26T01:25:57.206048+0000","last_peered":"2024-04-26T01:25:57.206048+0000","last_clean":"2024-04-26T01:25:57.206048+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:57.206048+0000","last_undegraded":"2024-04-26T01:25:57.206048+0000","last_fullsized":"2024-04-26T01:25:57.206048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:19.031090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:19.031090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.870416+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.507326+0000","last_change":"2024-04-26T01:25:22.810840+0000","last_active":"2024-04-26T01:25:57.507326+0000","last_peered":"2024-04-26T01:25:57.507326+0000","last_clean":"2024-04-26T01:25:57.507326+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:57.507326+0000","last_undegraded":"2024-04-26T01:25:57.507326+0000","last_fullsized":"2024-04-26T01:25:57.507326+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.810688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.810688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.060679+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066054+0000","last_change":"2024-04-26T01:25:10.080365+0000","last_active":"2024-04-26T01:25:57.066054+0000","last_peered":"2024-04-26T01:25:57.066054+0000","last_clean":"2024-04-26T01:25:57.066054+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:57.066054+0000","last_undegraded":"2024-04-26T01:25:57.066054+0000","last_fullsized":"2024-04-26T01:25:57.066054+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:10.080096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:10.080096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.198292+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":261,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206283+0000","last_change":"2024-04-26T01:25:20.498495+0000","last_active":"2024-04-26T01:25:57.206283+0000","last_peered":"2024-04-26T01:25:57.206283+0000","last_clean":"2024-04-26T01:25:57.206283+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:57.206283+0000","last_undegraded":"2024-04-26T01:25:57.206283+0000","last_fullsized":"2024-04-26T01:25:57.206283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:20.498167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:20.498167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:46.374939+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":251,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206973+0000","last_change":"2024-04-26T01:25:23.320622+0000","last_active":"2024-04-26T01:25:57.206973+0000","last_peered":"2024-04-26T01:25:57.206973+0000","last_clean":"2024-04-26T01:25:57.206973+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:57.206973+0000","last_undegraded":"2024-04-26T01:25:57.206973+0000","last_fullsized":"2024-04-26T01:25:57.206973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:23.320400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:23.320400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:26.196523+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":248,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206154+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:57.206154+0000","last_peered":"2024-04-26T01:25:57.206154+0000","last_clean":"2024-04-26T01:25:57.206154+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:57.206154+0000","last_undegraded":"2024-04-26T01:25:57.206154+0000","last_fullsized":"2024-04-26T01:25:57.206154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066020+0000","last_change":"2024-04-26T01:25:22.598146+0000","last_active":"2024-04-26T01:25:57.066020+0000","last_peered":"2024-04-26T01:25:57.066020+0000","last_clean":"2024-04-26T01:25:57.066020+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:57.066020+0000","last_undegraded":"2024-04-26T01:25:57.066020+0000","last_fullsized":"2024-04-26T01:25:57.066020+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.597918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.597918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:36.990896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":476,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206856+0000","last_change":"2024-04-26T01:25:29.907934+0000","last_active":"2024-04-26T01:25:57.206856+0000","last_peered":"2024-04-26T01:25:57.206856+0000","last_clean":"2024-04-26T01:25:57.206856+0000","last_became_active":"2024-04-26T01:25:29.907215+0000","last_became_peered":"2024-04-26T01:25:29.907215+0000","last_unstale":"2024-04-26T01:25:57.206856+0000","last_undegraded":"2024-04-26T01:25:57.206856+0000","last_fullsized":"2024-04-26T01:25:57.206856+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.311527+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":100727,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334836,"statfs":{"total":96636764160,"available":96598872064,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28740,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343100,"statfs":{"total":96636764160,"available":96607334400,"internally_reserved":0,"allocated":1753088,"data_stored":1827536,"data_compressed":397563,"data_compressed_allocated":540672,"data_compressed_original":1523712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28684,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343156,"statfs":{"total":96636764160,"available":96607391744,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1765376,"data_stored":1827536,"data_compressed":396125,"data_compressed_allocated":540672,"data_compressed_original":1511424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:26:02.408 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:26:02.698 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:02.698 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:26:02.714 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-26T01:26:01.855947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529812,"kb_used_data":26732,"kb_used_omap":24,"kb_used_meta":502823,"kb_avail":1509419628,"statfs":{"total":1546188226560,"available":1545645699072,"internally_reserved":0,"allocated":27373568,"data_stored":21835584,"data_compressed":5882800,"data_compressed_allocated":8077312,"data_compressed_original":17027072,"omap_allocated":25521,"internal_metadata":514890831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.817695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":247,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.207184+0000","last_change":"2024-04-26T01:25:21.992991+0000","last_active":"2024-04-26T01:25:57.207184+0000","last_peered":"2024-04-26T01:25:57.207184+0000","last_clean":"2024-04-26T01:25:57.207184+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:57.207184+0000","last_undegraded":"2024-04-26T01:25:57.207184+0000","last_fullsized":"2024-04-26T01:25:57.207184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:21.992665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:21.992665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:34.893460+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":249,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206048+0000","last_change":"2024-04-26T01:25:19.031279+0000","last_active":"2024-04-26T01:25:57.206048+0000","last_peered":"2024-04-26T01:25:57.206048+0000","last_clean":"2024-04-26T01:25:57.206048+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:57.206048+0000","last_undegraded":"2024-04-26T01:25:57.206048+0000","last_fullsized":"2024-04-26T01:25:57.206048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:19.031090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:19.031090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.870416+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.507326+0000","last_change":"2024-04-26T01:25:22.810840+0000","last_active":"2024-04-26T01:25:57.507326+0000","last_peered":"2024-04-26T01:25:57.507326+0000","last_clean":"2024-04-26T01:25:57.507326+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:57.507326+0000","last_undegraded":"2024-04-26T01:25:57.507326+0000","last_fullsized":"2024-04-26T01:25:57.507326+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.810688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.810688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.060679+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066054+0000","last_change":"2024-04-26T01:25:10.080365+0000","last_active":"2024-04-26T01:25:57.066054+0000","last_peered":"2024-04-26T01:25:57.066054+0000","last_clean":"2024-04-26T01:25:57.066054+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:57.066054+0000","last_undegraded":"2024-04-26T01:25:57.066054+0000","last_fullsized":"2024-04-26T01:25:57.066054+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:10.080096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:10.080096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.198292+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":261,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206283+0000","last_change":"2024-04-26T01:25:20.498495+0000","last_active":"2024-04-26T01:25:57.206283+0000","last_peered":"2024-04-26T01:25:57.206283+0000","last_clean":"2024-04-26T01:25:57.206283+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:57.206283+0000","last_undegraded":"2024-04-26T01:25:57.206283+0000","last_fullsized":"2024-04-26T01:25:57.206283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:20.498167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:20.498167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:46.374939+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":251,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206973+0000","last_change":"2024-04-26T01:25:23.320622+0000","last_active":"2024-04-26T01:25:57.206973+0000","last_peered":"2024-04-26T01:25:57.206973+0000","last_clean":"2024-04-26T01:25:57.206973+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:57.206973+0000","last_undegraded":"2024-04-26T01:25:57.206973+0000","last_fullsized":"2024-04-26T01:25:57.206973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:23.320400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:23.320400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:26.196523+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":248,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206154+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:57.206154+0000","last_peered":"2024-04-26T01:25:57.206154+0000","last_clean":"2024-04-26T01:25:57.206154+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:57.206154+0000","last_undegraded":"2024-04-26T01:25:57.206154+0000","last_fullsized":"2024-04-26T01:25:57.206154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066020+0000","last_change":"2024-04-26T01:25:22.598146+0000","last_active":"2024-04-26T01:25:57.066020+0000","last_peered":"2024-04-26T01:25:57.066020+0000","last_clean":"2024-04-26T01:25:57.066020+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:57.066020+0000","last_undegraded":"2024-04-26T01:25:57.066020+0000","last_fullsized":"2024-04-26T01:25:57.066020+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.597918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.597918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:36.990896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":476,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206856+0000","last_change":"2024-04-26T01:25:29.907934+0000","last_active":"2024-04-26T01:25:57.206856+0000","last_peered":"2024-04-26T01:25:57.206856+0000","last_clean":"2024-04-26T01:25:57.206856+0000","last_became_active":"2024-04-26T01:25:29.907215+0000","last_became_peered":"2024-04-26T01:25:29.907215+0000","last_unstale":"2024-04-26T01:25:57.206856+0000","last_undegraded":"2024-04-26T01:25:57.206856+0000","last_fullsized":"2024-04-26T01:25:57.206856+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.311527+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":100727,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334836,"statfs":{"total":96636764160,"available":96598872064,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28740,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343100,"statfs":{"total":96636764160,"available":96607334400,"internally_reserved":0,"allocated":1753088,"data_stored":1827536,"data_compressed":397563,"data_compressed_allocated":540672,"data_compressed_original":1523712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28684,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343156,"statfs":{"total":96636764160,"available":96607391744,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1765376,"data_stored":1827536,"data_compressed":396125,"data_compressed_allocated":540672,"data_compressed_original":1511424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:26:02.715 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T01:26:02.715 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:26:02.996 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:02.997 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:26:03.013 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-04-26T01:26:01.855947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529812,"kb_used_data":26732,"kb_used_omap":24,"kb_used_meta":502823,"kb_avail":1509419628,"statfs":{"total":1546188226560,"available":1545645699072,"internally_reserved":0,"allocated":27373568,"data_stored":21835584,"data_compressed":5882800,"data_compressed_allocated":8077312,"data_compressed_original":17027072,"omap_allocated":25521,"internal_metadata":514890831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.817695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":247,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.207184+0000","last_change":"2024-04-26T01:25:21.992991+0000","last_active":"2024-04-26T01:25:57.207184+0000","last_peered":"2024-04-26T01:25:57.207184+0000","last_clean":"2024-04-26T01:25:57.207184+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:25:57.207184+0000","last_undegraded":"2024-04-26T01:25:57.207184+0000","last_fullsized":"2024-04-26T01:25:57.207184+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:21.992665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:21.992665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:34.893460+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":249,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206048+0000","last_change":"2024-04-26T01:25:19.031279+0000","last_active":"2024-04-26T01:25:57.206048+0000","last_peered":"2024-04-26T01:25:57.206048+0000","last_clean":"2024-04-26T01:25:57.206048+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:25:57.206048+0000","last_undegraded":"2024-04-26T01:25:57.206048+0000","last_fullsized":"2024-04-26T01:25:57.206048+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:19.031090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:19.031090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.870416+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.507326+0000","last_change":"2024-04-26T01:25:22.810840+0000","last_active":"2024-04-26T01:25:57.507326+0000","last_peered":"2024-04-26T01:25:57.507326+0000","last_clean":"2024-04-26T01:25:57.507326+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:25:57.507326+0000","last_undegraded":"2024-04-26T01:25:57.507326+0000","last_fullsized":"2024-04-26T01:25:57.507326+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.810688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.810688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.060679+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":245,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066054+0000","last_change":"2024-04-26T01:25:10.080365+0000","last_active":"2024-04-26T01:25:57.066054+0000","last_peered":"2024-04-26T01:25:57.066054+0000","last_clean":"2024-04-26T01:25:57.066054+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:25:57.066054+0000","last_undegraded":"2024-04-26T01:25:57.066054+0000","last_fullsized":"2024-04-26T01:25:57.066054+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:10.080096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:10.080096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.198292+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":261,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206283+0000","last_change":"2024-04-26T01:25:20.498495+0000","last_active":"2024-04-26T01:25:57.206283+0000","last_peered":"2024-04-26T01:25:57.206283+0000","last_clean":"2024-04-26T01:25:57.206283+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:25:57.206283+0000","last_undegraded":"2024-04-26T01:25:57.206283+0000","last_fullsized":"2024-04-26T01:25:57.206283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:20.498167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:20.498167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:46.374939+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":251,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206973+0000","last_change":"2024-04-26T01:25:23.320622+0000","last_active":"2024-04-26T01:25:57.206973+0000","last_peered":"2024-04-26T01:25:57.206973+0000","last_clean":"2024-04-26T01:25:57.206973+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:25:57.206973+0000","last_undegraded":"2024-04-26T01:25:57.206973+0000","last_fullsized":"2024-04-26T01:25:57.206973+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:23.320400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:23.320400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:26.196523+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":248,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206154+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:25:57.206154+0000","last_peered":"2024-04-26T01:25:57.206154+0000","last_clean":"2024-04-26T01:25:57.206154+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:25:57.206154+0000","last_undegraded":"2024-04-26T01:25:57.206154+0000","last_fullsized":"2024-04-26T01:25:57.206154+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":250,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.066020+0000","last_change":"2024-04-26T01:25:22.598146+0000","last_active":"2024-04-26T01:25:57.066020+0000","last_peered":"2024-04-26T01:25:57.066020+0000","last_clean":"2024-04-26T01:25:57.066020+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:25:57.066020+0000","last_undegraded":"2024-04-26T01:25:57.066020+0000","last_fullsized":"2024-04-26T01:25:57.066020+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.597918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.597918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:36.990896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":476,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-04-26T01:25:57.206856+0000","last_change":"2024-04-26T01:25:29.907934+0000","last_active":"2024-04-26T01:25:57.206856+0000","last_peered":"2024-04-26T01:25:57.206856+0000","last_clean":"2024-04-26T01:25:57.206856+0000","last_became_active":"2024-04-26T01:25:29.907215+0000","last_became_peered":"2024-04-26T01:25:29.907215+0000","last_unstale":"2024-04-26T01:25:57.206856+0000","last_undegraded":"2024-04-26T01:25:57.206856+0000","last_fullsized":"2024-04-26T01:25:57.206856+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.311527+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":100727,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37004,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334836,"statfs":{"total":96636764160,"available":96598872064,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28740,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343100,"statfs":{"total":96636764160,"available":96607334400,"internally_reserved":0,"allocated":1753088,"data_stored":1827536,"data_compressed":397563,"data_compressed_allocated":540672,"data_compressed_original":1523712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32848,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338992,"statfs":{"total":96636764160,"available":96603127808,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313245,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313246,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37008,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334832,"statfs":{"total":96636764160,"available":96598867968,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28684,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343156,"statfs":{"total":96636764160,"available":96607391744,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313246,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313245,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32844,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1703936,"data_stored":1298608,"data_compressed":363508,"data_compressed_allocated":499712,"data_compressed_original":999424,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37072,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334768,"statfs":{"total":96636764160,"available":96598802432,"internally_reserved":0,"allocated":1765376,"data_stored":1827536,"data_compressed":396125,"data_compressed_allocated":540672,"data_compressed_original":1511424,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:26:03.015 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T01:26:03.293 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:03.293 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":144,"fsid":"ec0903d9-9475-4d3a-af50-1dbb367f53fb","created":"2024-04-26T01:23:35.072358+0000","modified":"2024-04-26T01:25:57.054171+0000","last_up_change":"2024-04-26T01:23:47.867650+0000","last_in_change":"2024-04-26T01:25:25.109979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T01:23:47.920762+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T01:23:50.025173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a5592650-ee45-45a1-94c4-487000bdb588","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6819","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6820","nonce":906916438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6824","nonce":906916438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6829","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6831","nonce":906916438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":906916438},{"type":"v1","addr":"172.21.15.37:6827","nonce":906916438}]},"public_addr":"172.21.15.37:6820/906916438","cluster_addr":"172.21.15.37:6824/906916438","heartbeat_back_addr":"172.21.15.37:6831/906916438","heartbeat_front_addr":"172.21.15.37:6827/906916438","state":["exists","up"]},{"osd":1,"uuid":"26af5691-fc53-4b20-9c35-012d299e978b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6813","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6816","nonce":1630954663}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6823","nonce":1630954663}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6831","nonce":1630954663}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":1630954663},{"type":"v1","addr":"172.21.15.45:6828","nonce":1630954663}]},"public_addr":"172.21.15.45:6816/1630954663","cluster_addr":"172.21.15.45:6823/1630954663","heartbeat_back_addr":"172.21.15.45:6831/1630954663","heartbeat_front_addr":"172.21.15.45:6828/1630954663","state":["exists","up"]},{"osd":2,"uuid":"ba3d860b-252b-42de-a951-99329413b1bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6801","nonce":2961312835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6803","nonce":2961312835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6807","nonce":2961312835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":2961312835},{"type":"v1","addr":"172.21.15.71:6805","nonce":2961312835}]},"public_addr":"172.21.15.71:6801/2961312835","cluster_addr":"172.21.15.71:6803/2961312835","heartbeat_back_addr":"172.21.15.71:6807/2961312835","heartbeat_front_addr":"172.21.15.71:6805/2961312835","state":["exists","up"]},{"osd":3,"uuid":"8294850e-c851-4be8-adb0-ffe9ef4fd827","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6801","nonce":1026435117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6806","nonce":1026435117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6821","nonce":1026435117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1026435117},{"type":"v1","addr":"172.21.15.181:6814","nonce":1026435117}]},"public_addr":"172.21.15.181:6801/1026435117","cluster_addr":"172.21.15.181:6806/1026435117","heartbeat_back_addr":"172.21.15.181:6821/1026435117","heartbeat_front_addr":"172.21.15.181:6814/1026435117","state":["exists","up"]},{"osd":4,"uuid":"601a444c-5cc7-4ae1-9cd3-379b6d7155da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6803","nonce":1575437742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6805","nonce":1575437742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6809","nonce":1575437742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":1575437742},{"type":"v1","addr":"172.21.15.37:6807","nonce":1575437742}]},"public_addr":"172.21.15.37:6803/1575437742","cluster_addr":"172.21.15.37:6805/1575437742","heartbeat_back_addr":"172.21.15.37:6809/1575437742","heartbeat_front_addr":"172.21.15.37:6807/1575437742","state":["exists","up"]},{"osd":5,"uuid":"6818ea43-206d-49df-9df8-53d9ff132f8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6801","nonce":3759230996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6803","nonce":3759230996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6807","nonce":3759230996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3759230996},{"type":"v1","addr":"172.21.15.45:6805","nonce":3759230996}]},"public_addr":"172.21.15.45:6801/3759230996","cluster_addr":"172.21.15.45:6803/3759230996","heartbeat_back_addr":"172.21.15.45:6807/3759230996","heartbeat_front_addr":"172.21.15.45:6805/3759230996","state":["exists","up"]},{"osd":6,"uuid":"7e47d5e9-00aa-428f-8b6c-256af34a821c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6809","nonce":76180415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6811","nonce":76180415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6815","nonce":76180415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":76180415},{"type":"v1","addr":"172.21.15.71:6813","nonce":76180415}]},"public_addr":"172.21.15.71:6809/76180415","cluster_addr":"172.21.15.71:6811/76180415","heartbeat_back_addr":"172.21.15.71:6815/76180415","heartbeat_front_addr":"172.21.15.71:6813/76180415","state":["exists","up"]},{"osd":7,"uuid":"0ed4ac32-3700-4879-abde-e664c307dcdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6807","nonce":4043929361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6812","nonce":4043929361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6823","nonce":4043929361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4043929361},{"type":"v1","addr":"172.21.15.181:6820","nonce":4043929361}]},"public_addr":"172.21.15.181:6807/4043929361","cluster_addr":"172.21.15.181:6812/4043929361","heartbeat_back_addr":"172.21.15.181:6823/4043929361","heartbeat_front_addr":"172.21.15.181:6820/4043929361","state":["exists","up"]},{"osd":8,"uuid":"94604649-8873-4348-a27c-47e6751274c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6811","nonce":4094482142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6813","nonce":4094482142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6817","nonce":4094482142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":4094482142},{"type":"v1","addr":"172.21.15.37:6815","nonce":4094482142}]},"public_addr":"172.21.15.37:6811/4094482142","cluster_addr":"172.21.15.37:6813/4094482142","heartbeat_back_addr":"172.21.15.37:6817/4094482142","heartbeat_front_addr":"172.21.15.37:6815/4094482142","state":["exists","up"]},{"osd":9,"uuid":"cf4159cb-22fe-49c9-b94c-36a6003e6c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6810","nonce":3448396888}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6815","nonce":3448396888}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6824","nonce":3448396888}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6817","nonce":3448396888},{"type":"v1","addr":"172.21.15.45:6819","nonce":3448396888}]},"public_addr":"172.21.15.45:6810/3448396888","cluster_addr":"172.21.15.45:6815/3448396888","heartbeat_back_addr":"172.21.15.45:6824/3448396888","heartbeat_front_addr":"172.21.15.45:6819/3448396888","state":["exists","up"]},{"osd":10,"uuid":"bf42aad2-b60d-45a4-adde-f258ce7086b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6817","nonce":896289472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6819","nonce":896289472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6823","nonce":896289472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":896289472},{"type":"v1","addr":"172.21.15.71:6821","nonce":896289472}]},"public_addr":"172.21.15.71:6817/896289472","cluster_addr":"172.21.15.71:6819/896289472","heartbeat_back_addr":"172.21.15.71:6823/896289472","heartbeat_front_addr":"172.21.15.71:6821/896289472","state":["exists","up"]},{"osd":11,"uuid":"1db3483a-2bc5-4cb5-9034-224068e68cdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6825","nonce":647568313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6827","nonce":647568313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6831","nonce":647568313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":647568313},{"type":"v1","addr":"172.21.15.181:6829","nonce":647568313}]},"public_addr":"172.21.15.181:6825/647568313","cluster_addr":"172.21.15.181:6827/647568313","heartbeat_back_addr":"172.21.15.181:6831/647568313","heartbeat_front_addr":"172.21.15.181:6829/647568313","state":["exists","up"]},{"osd":12,"uuid":"970dc59e-69c3-4a15-ada5-569b3ee520b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6821","nonce":3336489453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6823","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6825","nonce":3336489453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6833","nonce":3336489453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":3336489453},{"type":"v1","addr":"172.21.15.37:6830","nonce":3336489453}]},"public_addr":"172.21.15.37:6821/3336489453","cluster_addr":"172.21.15.37:6825/3336489453","heartbeat_back_addr":"172.21.15.37:6833/3336489453","heartbeat_front_addr":"172.21.15.37:6830/3336489453","state":["exists","up"]},{"osd":13,"uuid":"cae6ef09-e4e5-4d6c-8934-065720075a50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6809","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6811","nonce":1608953722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6818","nonce":1608953722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6827","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6829","nonce":1608953722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6821","nonce":1608953722},{"type":"v1","addr":"172.21.15.45:6825","nonce":1608953722}]},"public_addr":"172.21.15.45:6811/1608953722","cluster_addr":"172.21.15.45:6818/1608953722","heartbeat_back_addr":"172.21.15.45:6829/1608953722","heartbeat_front_addr":"172.21.15.45:6825/1608953722","state":["exists","up"]},{"osd":14,"uuid":"02493b55-1966-4731-8bba-891cfb8f2244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6825","nonce":3719375992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6827","nonce":3719375992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6831","nonce":3719375992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3719375992},{"type":"v1","addr":"172.21.15.71:6829","nonce":3719375992}]},"public_addr":"172.21.15.71:6825/3719375992","cluster_addr":"172.21.15.71:6827/3719375992","heartbeat_back_addr":"172.21.15.71:6831/3719375992","heartbeat_front_addr":"172.21.15.71:6829/3719375992","state":["exists","up"]},{"osd":15,"uuid":"93d2b85d-c322-44b6-aa6a-c5da975ba45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6805","nonce":2891128021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6811","nonce":2891128021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6817","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6819","nonce":2891128021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2891128021},{"type":"v1","addr":"172.21.15.181:6815","nonce":2891128021}]},"public_addr":"172.21.15.181:6805/2891128021","cluster_addr":"172.21.15.181:6811/2891128021","heartbeat_back_addr":"172.21.15.181:6819/2891128021","heartbeat_front_addr":"172.21.15.181:6815/2891128021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.150989+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:50.485009+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:55.015898+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.670769+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:05.579754+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:08.910977+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:51.291925+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:51.369682+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:54.515953+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:14.187344+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:13.456713+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:02.015135+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.667171+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:25:04.495812+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:48.329599+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T01:24:47.498480+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":25,"length":1},{"begin":27,"length":2},{"begin":33,"length":1},{"begin":35,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T01:26:04.312 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T01:26:04.312 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:04.476 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:04.476 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:04.476 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:04.493 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T01:26:04.787 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T01:26:04.805 INFO:tasks.ceph:Scrubbing osd.4 2024-04-26T01:26:04.805 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:04.958 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:04.958 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:04.958 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:04.974 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-26T01:26:05.271 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 4 to deep-scrub 2024-04-26T01:26:05.291 INFO:tasks.ceph:Scrubbing osd.8 2024-04-26T01:26:05.291 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:05.442 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:05.442 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:05.442 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:05.459 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-04-26T01:26:05.746 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 8 to deep-scrub 2024-04-26T01:26:05.764 INFO:tasks.ceph:Scrubbing osd.12 2024-04-26T01:26:05.764 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:05.902 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:05.902 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:05.902 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:05.917 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-04-26T01:26:06.207 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 12 to deep-scrub 2024-04-26T01:26:06.225 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T01:26:06.226 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:06.369 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:06.369 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:06.369 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:06.385 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T01:26:06.678 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T01:26:06.698 INFO:tasks.ceph:Scrubbing osd.5 2024-04-26T01:26:06.698 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:06.858 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:06.858 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:06.858 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:06.875 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-26T01:26:07.168 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 5 to deep-scrub 2024-04-26T01:26:07.187 INFO:tasks.ceph:Scrubbing osd.9 2024-04-26T01:26:07.187 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:07.345 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:07.345 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:07.345 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:07.361 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-04-26T01:26:07.652 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 9 to deep-scrub 2024-04-26T01:26:07.672 INFO:tasks.ceph:Scrubbing osd.13 2024-04-26T01:26:07.672 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:07.832 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:07.832 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:07.832 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:07.849 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-04-26T01:26:08.147 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 13 to deep-scrub 2024-04-26T01:26:08.166 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T01:26:08.166 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:08.317 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:08.317 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:08.317 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:08.333 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T01:26:08.633 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T01:26:08.653 INFO:tasks.ceph:Scrubbing osd.6 2024-04-26T01:26:08.653 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:08.808 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:08.808 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:08.808 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:08.825 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-26T01:26:09.120 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 6 to deep-scrub 2024-04-26T01:26:09.138 INFO:tasks.ceph:Scrubbing osd.10 2024-04-26T01:26:09.138 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:09.290 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:09.290 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:09.290 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:09.307 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-04-26T01:26:09.609 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 10 to deep-scrub 2024-04-26T01:26:09.628 INFO:tasks.ceph:Scrubbing osd.14 2024-04-26T01:26:09.628 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:09.775 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:09.775 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:09.775 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:09.791 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-04-26T01:26:10.082 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 14 to deep-scrub 2024-04-26T01:26:10.102 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T01:26:10.102 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:10.266 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:10.267 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:10.267 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:10.284 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T01:26:10.567 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T01:26:10.585 INFO:tasks.ceph:Scrubbing osd.7 2024-04-26T01:26:10.586 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:10.743 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:10.744 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:10.744 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:10.760 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-26T01:26:11.049 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 7 to deep-scrub 2024-04-26T01:26:11.067 INFO:tasks.ceph:Scrubbing osd.11 2024-04-26T01:26:11.068 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:11.216 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:11.216 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:11.216 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:11.231 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-04-26T01:26:11.516 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 11 to deep-scrub 2024-04-26T01:26:11.536 INFO:tasks.ceph:Scrubbing osd.15 2024-04-26T01:26:11.536 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-04-26T01:26:11.684 INFO:teuthology.orchestra.run.smithi037.stdout:{ 2024-04-26T01:26:11.685 INFO:teuthology.orchestra.run.smithi037.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T01:26:11.685 INFO:teuthology.orchestra.run.smithi037.stdout:} 2024-04-26T01:26:11.701 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-04-26T01:26:11.989 INFO:teuthology.orchestra.run.smithi037.stderr:instructed osd(s) 15 to deep-scrub 2024-04-26T01:26:12.007 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:26:12.301 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:12.301 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:26:12.320 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-04-26T01:26:11.858496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529988,"kb_used_data":26948,"kb_used_omap":24,"kb_used_meta":502823,"kb_avail":1509419452,"statfs":{"total":1546188226560,"available":1545645518848,"internally_reserved":0,"allocated":27594752,"data_stored":22009752,"data_compressed":5936170,"data_compressed_allocated":8151040,"data_compressed_original":17174528,"omap_allocated":25521,"internal_metadata":514890831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.722475"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":250,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-04-26T01:26:06.360008+0000","last_change":"2024-04-26T01:25:21.992991+0000","last_active":"2024-04-26T01:26:06.360008+0000","last_peered":"2024-04-26T01:26:06.360008+0000","last_clean":"2024-04-26T01:26:06.360008+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:26:06.360008+0000","last_undegraded":"2024-04-26T01:26:06.360008+0000","last_fullsized":"2024-04-26T01:26:06.360008+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:21.992665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:21.992665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:34.893460+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":263,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-04-26T01:26:08.560109+0000","last_change":"2024-04-26T01:26:06.276642+0000","last_active":"2024-04-26T01:26:08.560109+0000","last_peered":"2024-04-26T01:26:08.560109+0000","last_clean":"2024-04-26T01:26:08.560109+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:26:08.560109+0000","last_undegraded":"2024-04-26T01:26:08.560109+0000","last_fullsized":"2024-04-26T01:26:08.560109+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:06.276495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:06.276495+0000","last_clean_scrub_stamp":"2024-04-26T01:26:06.276495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:22.220569+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":252,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-04-26T01:26:06.257617+0000","last_change":"2024-04-26T01:25:22.810840+0000","last_active":"2024-04-26T01:26:06.257617+0000","last_peered":"2024-04-26T01:26:06.257617+0000","last_clean":"2024-04-26T01:26:06.257617+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:26:06.257617+0000","last_undegraded":"2024-04-26T01:26:06.257617+0000","last_fullsized":"2024-04-26T01:26:06.257617+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.810688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.810688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:31.060679+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":256,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-04-26T01:26:06.238301+0000","last_change":"2024-04-26T01:26:06.190382+0000","last_active":"2024-04-26T01:26:06.238301+0000","last_peered":"2024-04-26T01:26:06.238301+0000","last_clean":"2024-04-26T01:26:06.238301+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:26:06.238301+0000","last_undegraded":"2024-04-26T01:26:06.238301+0000","last_fullsized":"2024-04-26T01:26:06.238301+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:06.190077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:06.190077+0000","last_clean_scrub_stamp":"2024-04-26T01:26:06.190077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:25.542753+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":263,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-04-26T01:26:06.258728+0000","last_change":"2024-04-26T01:25:20.498495+0000","last_active":"2024-04-26T01:26:06.258728+0000","last_peered":"2024-04-26T01:26:06.258728+0000","last_clean":"2024-04-26T01:26:06.258728+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:26:06.258728+0000","last_undegraded":"2024-04-26T01:26:06.258728+0000","last_fullsized":"2024-04-26T01:26:06.258728+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:20.498167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:20.498167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:46.374939+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":256,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-04-26T01:26:08.196858+0000","last_change":"2024-04-26T01:25:23.320622+0000","last_active":"2024-04-26T01:26:08.196858+0000","last_peered":"2024-04-26T01:26:08.196858+0000","last_clean":"2024-04-26T01:26:08.196858+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:26:08.196858+0000","last_undegraded":"2024-04-26T01:26:08.196858+0000","last_fullsized":"2024-04-26T01:26:08.196858+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:23.320400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:23.320400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":250,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-04-26T01:26:06.258728+0000","last_change":"2024-04-26T01:24:54.798203+0000","last_active":"2024-04-26T01:26:06.258728+0000","last_peered":"2024-04-26T01:26:06.258728+0000","last_clean":"2024-04-26T01:26:06.258728+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:26:06.258728+0000","last_undegraded":"2024-04-26T01:26:06.258728+0000","last_fullsized":"2024-04-26T01:26:06.258728+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:24:54.797961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:24:54.797961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:26:24.353690+0000","scrub_duration":463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":253,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-04-26T01:26:06.358251+0000","last_change":"2024-04-26T01:25:22.598146+0000","last_active":"2024-04-26T01:26:06.358251+0000","last_peered":"2024-04-26T01:26:06.358251+0000","last_clean":"2024-04-26T01:26:06.358251+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:26:06.358251+0000","last_undegraded":"2024-04-26T01:26:06.358251+0000","last_fullsized":"2024-04-26T01:26:06.358251+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:25:22.597918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:50.882551+0000","last_clean_scrub_stamp":"2024-04-26T01:25:22.597918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"34'157","reported_seq":481,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-04-26T01:26:08.196777+0000","last_change":"2024-04-26T01:25:29.907934+0000","last_active":"2024-04-26T01:26:08.196777+0000","last_peered":"2024-04-26T01:26:08.196777+0000","last_clean":"2024-04-26T01:26:08.196777+0000","last_became_active":"2024-04-26T01:25:29.907215+0000","last_became_peered":"2024-04-26T01:25:29.907215+0000","last_unstale":"2024-04-26T01:26:08.196777+0000","last_undegraded":"2024-04-26T01:26:08.196777+0000","last_fullsized":"2024-04-26T01:26:08.196777+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'157","last_scrub_stamp":"2024-04-26T01:24:56.353425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:23:48.870422+0000","last_clean_scrub_stamp":"2024-04-26T01:24:56.353425+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":100727,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1728512,"data_stored":1317960,"data_compressed":369438,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1728512,"data_stored":1317960,"data_compressed":369438,"data_compressed_allocated":507904,"data_compressed_original":1015808,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37016,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334824,"statfs":{"total":96636764160,"available":96598859776,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313248,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28744,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343096,"statfs":{"total":96636764160,"available":96607330304,"internally_reserved":0,"allocated":1765376,"data_stored":1837212,"data_compressed":400528,"data_compressed_allocated":544768,"data_compressed_original":1531904,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313247,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37016,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334824,"statfs":{"total":96636764160,"available":96598859776,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313248,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28696,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343144,"statfs":{"total":96636764160,"available":96607379456,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313249,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":1308284,"data_compressed":366473,"data_compressed_allocated":503808,"data_compressed_original":1007616,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313248,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1777664,"data_stored":1837212,"data_compressed":399090,"data_compressed_allocated":544768,"data_compressed_original":1519616,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:26:12.323 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-26T01:25:21.992665+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=25, tm_sec=21, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=26, tm_sec=3, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:26:12.324 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-26T01:25:22.810688+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=25, tm_sec=22, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=26, tm_sec=3, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:26:12.324 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-26T01:25:20.498167+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=25, tm_sec=20, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=26, tm_sec=3, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:26:12.324 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-26T01:25:23.320400+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=25, tm_sec=23, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=26, tm_sec=3, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:26:12.324 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-26T01:24:54.797961+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=26, tm_sec=3, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:26:12.324 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-26T01:25:22.597918+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=25, tm_sec=22, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=26, tm_sec=3, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:26:12.324 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-26T01:24:56.353425+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=24, tm_sec=56, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=1, tm_min=26, tm_sec=3, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T01:26:12.324 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T01:26:32.326 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T01:26:32.640 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-26T01:26:32.641 INFO:teuthology.orchestra.run.smithi037.stderr:dumped all 2024-04-26T01:26:32.659 INFO:teuthology.orchestra.run.smithi037.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-04-26T01:26:31.864504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":531392,"kb_used_data":28340,"kb_used_omap":24,"kb_used_meta":502823,"kb_avail":1509418048,"statfs":{"total":1546188226560,"available":1545644081152,"internally_reserved":0,"allocated":29020160,"data_stored":23132968,"data_compressed":6280162,"data_compressed_allocated":8626176,"data_compressed_original":18124800,"omap_allocated":25521,"internal_metadata":514890831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002486"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":273,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-26T01:26:18.877310+0000","last_change":"2024-04-26T01:26:14.098836+0000","last_active":"2024-04-26T01:26:18.877310+0000","last_peered":"2024-04-26T01:26:18.877310+0000","last_clean":"2024-04-26T01:26:18.877310+0000","last_became_active":"2024-04-26T01:23:51.915505+0000","last_became_peered":"2024-04-26T01:23:51.915505+0000","last_unstale":"2024-04-26T01:26:18.877310+0000","last_undegraded":"2024-04-26T01:26:18.877310+0000","last_fullsized":"2024-04-26T01:26:18.877310+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:14.098568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:14.098568+0000","last_clean_scrub_stamp":"2024-04-26T01:26:14.098568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:35.836835+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13],"acting":[15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":277,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-26T01:26:26.929577+0000","last_change":"2024-04-26T01:26:06.276642+0000","last_active":"2024-04-26T01:26:26.929577+0000","last_peered":"2024-04-26T01:26:26.929577+0000","last_clean":"2024-04-26T01:26:26.929577+0000","last_became_active":"2024-04-26T01:23:51.915168+0000","last_became_peered":"2024-04-26T01:23:51.915168+0000","last_unstale":"2024-04-26T01:26:26.929577+0000","last_undegraded":"2024-04-26T01:26:26.929577+0000","last_fullsized":"2024-04-26T01:26:26.929577+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:06.276495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:06.276495+0000","last_clean_scrub_stamp":"2024-04-26T01:26:06.276495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:22.220569+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":275,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-26T01:26:19.958768+0000","last_change":"2024-04-26T01:26:09.803789+0000","last_active":"2024-04-26T01:26:19.958768+0000","last_peered":"2024-04-26T01:26:19.958768+0000","last_clean":"2024-04-26T01:26:19.958768+0000","last_became_active":"2024-04-26T01:23:51.914303+0000","last_became_peered":"2024-04-26T01:23:51.914303+0000","last_unstale":"2024-04-26T01:26:19.958768+0000","last_undegraded":"2024-04-26T01:26:19.958768+0000","last_fullsized":"2024-04-26T01:26:19.958768+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:09.803565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:09.803565+0000","last_clean_scrub_stamp":"2024-04-26T01:26:09.803565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:13.446153+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":271,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-26T01:26:28.276016+0000","last_change":"2024-04-26T01:26:06.190382+0000","last_active":"2024-04-26T01:26:28.276016+0000","last_peered":"2024-04-26T01:26:28.276016+0000","last_clean":"2024-04-26T01:26:28.276016+0000","last_became_active":"2024-04-26T01:23:52.270106+0000","last_became_peered":"2024-04-26T01:23:52.270106+0000","last_unstale":"2024-04-26T01:26:28.276016+0000","last_undegraded":"2024-04-26T01:26:28.276016+0000","last_fullsized":"2024-04-26T01:26:28.276016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:06.190077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:06.190077+0000","last_clean_scrub_stamp":"2024-04-26T01:26:06.190077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:25.542753+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":286,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-26T01:26:19.114937+0000","last_change":"2024-04-26T01:26:12.147339+0000","last_active":"2024-04-26T01:26:19.114937+0000","last_peered":"2024-04-26T01:26:19.114937+0000","last_clean":"2024-04-26T01:26:19.114937+0000","last_became_active":"2024-04-26T01:25:02.919904+0000","last_became_peered":"2024-04-26T01:25:02.919904+0000","last_unstale":"2024-04-26T01:26:19.114937+0000","last_undegraded":"2024-04-26T01:26:19.114937+0000","last_fullsized":"2024-04-26T01:26:19.114937+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:12.147013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:12.147013+0000","last_clean_scrub_stamp":"2024-04-26T01:26:12.147013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:35.179944+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,12],"acting":[3,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":278,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-26T01:26:27.663889+0000","last_change":"2024-04-26T01:26:09.321624+0000","last_active":"2024-04-26T01:26:27.663889+0000","last_peered":"2024-04-26T01:26:27.663889+0000","last_clean":"2024-04-26T01:26:27.663889+0000","last_became_active":"2024-04-26T01:23:51.905598+0000","last_became_peered":"2024-04-26T01:23:51.905598+0000","last_unstale":"2024-04-26T01:26:27.663889+0000","last_undegraded":"2024-04-26T01:26:27.663889+0000","last_fullsized":"2024-04-26T01:26:27.663889+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:09.321379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:09.321379+0000","last_clean_scrub_stamp":"2024-04-26T01:26:09.321379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:22.266300+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":273,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-04-26T01:26:19.114994+0000","last_change":"2024-04-26T01:26:11.268461+0000","last_active":"2024-04-26T01:26:19.114994+0000","last_peered":"2024-04-26T01:26:19.114994+0000","last_clean":"2024-04-26T01:26:19.114994+0000","last_became_active":"2024-04-26T01:23:52.211139+0000","last_became_peered":"2024-04-26T01:23:52.211139+0000","last_unstale":"2024-04-26T01:26:19.114994+0000","last_undegraded":"2024-04-26T01:26:19.114994+0000","last_fullsized":"2024-04-26T01:26:19.114994+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:11.268223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:11.268223+0000","last_clean_scrub_stamp":"2024-04-26T01:26:11.268223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:11.430017+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":276,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-26T01:26:26.929913+0000","last_change":"2024-04-26T01:26:08.008887+0000","last_active":"2024-04-26T01:26:26.929913+0000","last_peered":"2024-04-26T01:26:26.929913+0000","last_clean":"2024-04-26T01:26:26.929913+0000","last_became_active":"2024-04-26T01:23:51.913776+0000","last_became_peered":"2024-04-26T01:23:51.913776+0000","last_unstale":"2024-04-26T01:26:26.929913+0000","last_undegraded":"2024-04-26T01:26:26.929913+0000","last_fullsized":"2024-04-26T01:26:26.929913+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T01:26:08.008503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T01:26:08.008503+0000","last_clean_scrub_stamp":"2024-04-26T01:26:08.008503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:18.315015+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,15],"acting":[5,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"146'159","reported_seq":505,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-04-26T01:26:27.763927+0000","last_change":"2024-04-26T01:26:10.583580+0000","last_active":"2024-04-26T01:26:27.763927+0000","last_peered":"2024-04-26T01:26:27.763927+0000","last_clean":"2024-04-26T01:26:27.763927+0000","last_became_active":"2024-04-26T01:25:29.907215+0000","last_became_peered":"2024-04-26T01:25:29.907215+0000","last_unstale":"2024-04-26T01:26:27.763927+0000","last_undegraded":"2024-04-26T01:26:27.763927+0000","last_fullsized":"2024-04-26T01:26:27.763927+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'159","last_scrub_stamp":"2024-04-26T01:26:10.583308+0000","last_deep_scrub":"146'159","last_deep_scrub_stamp":"2024-04-26T01:26:10.583308+0000","last_clean_scrub_stamp":"2024-04-26T01:26:10.583308+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:27:28.249392+0000","scrub_duration":307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2],"acting":[13,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":118,"num_write":203,"num_write_kb":3460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":1586784,"data_compressed":100727,"data_compressed_allocated":122880,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":12,"up_from":22,"seq":94489280549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32952,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1814528,"data_stored":1385742,"data_compressed":390195,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32952,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1814528,"data_stored":1385742,"data_compressed":390195,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37104,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334736,"statfs":{"total":96636764160,"available":96598769664,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,5,6,7,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32952,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1814528,"data_stored":1385742,"data_compressed":390195,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,5,6,7,8,9,10,11,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313252,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28840,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343000,"statfs":{"total":96636764160,"available":96607232000,"internally_reserved":0,"allocated":1863680,"data_stored":1914670,"data_compressed":424250,"data_compressed_allocated":577536,"data_compressed_original":1597440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,6,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32944,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338896,"statfs":{"total":96636764160,"available":96603029504,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":21,"seq":90194313252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32944,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338896,"statfs":{"total":96636764160,"available":96603029504,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313251,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37100,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334740,"statfs":{"total":96636764160,"available":96598773760,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313252,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,4,5,7,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":21,"seq":90194313252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28780,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343060,"statfs":{"total":96636764160,"available":96607293440,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,6,8,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":21,"seq":90194313252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32952,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338888,"statfs":{"total":96636764160,"available":96603021312,"internally_reserved":0,"allocated":1814528,"data_stored":1385742,"data_compressed":390195,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":21,"seq":90194313251,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,3,4,5,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":21,"seq":90194313253,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":1376066,"data_compressed":387232,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":21,"seq":90194313252,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37172,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334668,"statfs":{"total":96636764160,"available":96598700032,"internally_reserved":0,"allocated":1875968,"data_stored":1914670,"data_compressed":422812,"data_compressed_allocated":577536,"data_compressed_original":1585152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,3,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":34055,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32617,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T01:26:32.660 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T01:26:32.972 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T01:26:32.972 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T01:26:32.972 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T01:26:32.973 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:33.059 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T01:26:33.060 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-26T01:26:33.060 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:33.137 INFO:tasks.ceph.osd.4:Stopped 2024-04-26T01:26:33.137 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-04-26T01:26:33.137 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:33.230 INFO:tasks.ceph.osd.8:Stopped 2024-04-26T01:26:33.230 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-04-26T01:26:33.230 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:33.307 INFO:tasks.ceph.osd.12:Stopped 2024-04-26T01:26:33.307 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T01:26:33.307 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:33.399 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T01:26:33.399 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-26T01:26:33.399 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:33.472 INFO:tasks.ceph.osd.5:Stopped 2024-04-26T01:26:33.473 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-04-26T01:26:33.473 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:38.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 is failed for ~0s 2024-04-26T01:26:39.575 INFO:tasks.ceph.osd.9:Stopped 2024-04-26T01:26:39.576 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-04-26T01:26:39.576 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:39.667 INFO:tasks.ceph.osd.13:Stopped 2024-04-26T01:26:39.667 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T01:26:39.667 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:39.757 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T01:26:39.757 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-26T01:26:39.757 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:39.853 INFO:tasks.ceph.osd.6:Stopped 2024-04-26T01:26:39.853 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-04-26T01:26:39.853 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:44.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 is failed for ~0s 2024-04-26T01:26:44.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.9 has been restored 2024-04-26T01:26:45.954 INFO:tasks.ceph.osd.10:Stopped 2024-04-26T01:26:45.954 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-04-26T01:26:45.955 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:46.041 INFO:tasks.ceph.osd.14:Stopped 2024-04-26T01:26:46.042 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T01:26:46.042 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:46.139 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T01:26:46.139 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-26T01:26:46.139 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:46.231 INFO:tasks.ceph.osd.7:Stopped 2024-04-26T01:26:46.232 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-04-26T01:26:46.232 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:49.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 is failed for ~0s 2024-04-26T01:26:49.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.10 has been restored 2024-04-26T01:26:52.335 INFO:tasks.ceph.osd.11:Stopped 2024-04-26T01:26:52.335 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-04-26T01:26:52.335 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:52.390 INFO:tasks.ceph.osd.15:Stopped 2024-04-26T01:26:52.391 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T01:26:52.391 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-26T01:26:52.391 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:52.421 INFO:tasks.ceph.mgr.y:Stopped 2024-04-26T01:26:52.421 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T01:26:52.421 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:52.451 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T01:26:52.452 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T01:26:52.452 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T01:26:52.452 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:52.485 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T01:26:52.485 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T01:26:52.486 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:52.505 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T01:26:52.506 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T01:26:52.506 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T01:26:52.531 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T01:26:52.531 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T01:26:52.531 DEBUG:teuthology.orchestra.run.smithi037:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-04-26T01:26:52.548 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:26:52.548 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T01:26:53.017 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:26:53.017 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-26T01:26:53.261 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:26:53.262 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-04-26T01:26:53.419 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:26:53.419 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-04-26T01:26:53.644 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:26:53.645 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T01:26:53.801 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:26:53.802 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-26T01:26:53.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:26:53.996 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-04-26T01:26:54.237 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:26:54.238 DEBUG:teuthology.orchestra.run.smithi045:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-04-26T01:26:54.544 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:26:54.544 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T01:26:54.775 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:26:54.775 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-26T01:26:54.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 has been restored 2024-04-26T01:26:54.984 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:26:54.984 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-04-26T01:26:55.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:26:55.191 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-04-26T01:26:55.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:26:55.401 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T01:26:55.576 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:26:55.576 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-26T01:26:55.827 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:26:55.827 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-04-26T01:26:56.019 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:26:56.019 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-04-26T01:26:56.194 INFO:tasks.ceph:Archiving mon data... 2024-04-26T01:26:56.195 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/data/mon.a.tgz 2024-04-26T01:26:56.196 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-04-26T01:26:56.202 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.YSJNU7hkOY 2024-04-26T01:26:56.202 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.YSJNU7hkOY 2024-04-26T01:26:56.360 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.YSJNU7hkOY 2024-04-26T01:26:56.421 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.YSJNU7hkOY is 436KB 2024-04-26T01:26:56.483 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.YSJNU7hkOY 2024-04-26T01:26:56.489 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/data/mon.b.tgz 2024-04-26T01:26:56.489 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-04-26T01:26:56.495 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.MeYtnTHMJy 2024-04-26T01:26:56.495 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.MeYtnTHMJy 2024-04-26T01:26:56.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.MeYtnTHMJy 2024-04-26T01:26:56.754 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.MeYtnTHMJy is 632KB 2024-04-26T01:26:56.819 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.MeYtnTHMJy 2024-04-26T01:26:56.824 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/data/mon.c.tgz 2024-04-26T01:26:56.824 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-04-26T01:26:56.830 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.Q2c7K9YHmy 2024-04-26T01:26:56.831 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Q2c7K9YHmy 2024-04-26T01:26:57.026 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.Q2c7K9YHmy 2024-04-26T01:26:57.088 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.Q2c7K9YHmy is 632KB 2024-04-26T01:26:57.150 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.Q2c7K9YHmy 2024-04-26T01:26:57.157 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T01:26:57.158 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:26:57.160 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:26:57.163 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:26:57.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T01:26:57.220 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:26:57.226 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:26:57.260 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:26:57.266 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T01:26:57.266 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T01:26:57.267 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi037/crash 2024-04-26T01:26:57.268 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:26:57.282 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi045/crash 2024-04-26T01:26:57.283 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:26:57.298 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi071/crash 2024-04-26T01:26:57.299 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:26:57.317 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi181/crash 2024-04-26T01:26:57.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T01:26:57.331 INFO:tasks.ceph:Compressing logs... 2024-04-26T01:26:57.331 DEBUG:teuthology.orchestra.run.smithi037:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:26:57.335 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:26:57.342 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:26:57.356 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31282.log 2024-04-26T01:26:57.357 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33201.log 2024-04-26T01:26:57.357 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31282.log.gz 2024-04-26T01:26:57.357 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28880.log 2024-04-26T01:26:57.358 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33201.log.gz 2024-04-26T01:26:57.358 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32161.log 2024-04-26T01:26:57.358 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29286.log 2024-04-26T01:26:57.359 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28880.log: /var/log/ceph/ceph-client.admin.32161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28716.log 2024-04-26T01:26:57.359 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28880.log.gz 2024-04-26T01:26:57.359 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29286.log.gz 2024-04-26T01:26:57.359 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32161.log.gz 2024-04-26T01:26:57.359 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29304.log 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36510.log 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28716.log.gz 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29304.log.gz 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28871.log 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.36510.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36069.log 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36510.log.gz 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28871.log.gz 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29302.log 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29796.log 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36069.log.gz 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29302.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29302.log.gz 2024-04-26T01:26:57.364 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29796.log.gz 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34139.log 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34404.log 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34139.log.gz 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36025.log 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.34404.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29295.log 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34404.log.gz 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32537.log 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36025.log.gz 2024-04-26T01:26:57.365 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29295.log.gz 2024-04-26T01:26:57.366 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37213.log 2024-04-26T01:26:57.367 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-26T01:26:57.370 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-osd.9.log: /var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T01:26:57.371 INFO:teuthology.orchestra.run.smithi045.stderr: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34976.log 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32537.log.gz 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35637.log 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37213.log.gz 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35080.log 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34976.log.gz 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35181.log 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35637.log.gz 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29766.log 2024-04-26T01:26:57.386 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35080.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35181.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20183.log 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30258.log 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29766.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.tmp-client.admin.20183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35203.log 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20183.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32298.log 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30258.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35208.log 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35203.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37369.log 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32298.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35208.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.28794.log -- replaced with /var/log/ceph/ceph-client.admin.35208.log.gz 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T01:26:57.387 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37369.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34690.log 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28794.log.gz 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33737.log 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29873.log 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34690.log.gz 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33737.log.gz 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32988.log 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29873.log.gz 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32406.log 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35196.log 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32988.log.gz 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32406.log.gz 2024-04-26T01:26:57.388 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30123.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32907.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35196.log.gz 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30123.log.gz 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34243.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35132.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32907.log: /var/log/ceph/ceph-client.admin.34243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32907.log.gz 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34243.log.gz 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35132.log.gz 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30680.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33347.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34768.log 2024-04-26T01:26:57.389 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30680.log.gz 2024-04-26T01:26:57.390 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37109.log 2024-04-26T01:26:57.390 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33347.log.gz 2024-04-26T01:26:57.390 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-04-26T01:26:57.390 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34768.log.gz 2024-04-26T01:26:57.390 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37109.log.gz 2024-04-26T01:26:57.390 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31121.log 2024-04-26T01:26:57.390 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37445.log 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31312.log: /var/log/ceph/ceph-client.admin.31121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36406.log 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31121.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37445.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37445.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31008.log 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34612.log 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36406.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34300.log 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31008.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34612.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32136.log 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36822.log 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34300.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32136.log.gz 2024-04-26T01:26:57.391 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32568.log 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36123.log 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36822.log.gz 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32568.log.gz 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36484.log 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36123.log.gz 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29257.log 2024-04-26T01:26:57.392 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37291.log 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36484.log.gz 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29257.log.gz 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35182.log 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37291.log.gz 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35182.log.gz 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28878.log 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34794.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28878.log.gz 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36562.log 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29265.log 2024-04-26T01:26:57.393 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36562.log.gz 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30318.log 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30445.log 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29265.log: /var/log/ceph/ceph-client.admin.30318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29265.log.gz 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37317.log 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30318.log.gz 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30445.log.gz 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31677.log 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30772.log 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37317.log.gz 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29280.log 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31677.log.gz 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30772.log.gz 2024-04-26T01:26:57.394 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33472.log 2024-04-26T01:26:57.399 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.10.log: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T01:26:57.400 INFO:teuthology.orchestra.run.smithi071.stderr: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T01:26:57.414 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25263.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24967.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25087.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25263.log.gz 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.24967.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24967.log.gz 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25087.log.gz 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25190.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25015.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-client.admin.25190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25039.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25190.log.gz 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25015.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-26T01:26:57.415 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25015.log.gz 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25039.log.gz 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24919.log 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.24919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24919.log.gz 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25118.log 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.28752.log 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25118.log.gz 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24943.log 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.0.28752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.28752.log.gz 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.24943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24991.log 0.0% 2024-04-26T01:26:57.416 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24943.log.gz 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25287.log 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.24991.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.24991.log.gz 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25214.log 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25287.log.gz 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25166.log 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25142.log 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25214.log.gz 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25166.log.gz 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25238.log 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25063.log 2024-04-26T01:26:57.417 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25142.log: 0.0%/var/log/ceph/ceph-client.admin.25238.log: -- replaced with /var/log/ceph/ceph-client.admin.25142.log.gz 2024-04-26T01:26:57.418 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25238.log.gz 2024-04-26T01:26:57.418 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.25063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25063.log.gz 2024-04-26T01:26:57.418 INFO:teuthology.orchestra.run.smithi181.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-26T01:26:57.478 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32662.log 2024-04-26T01:26:57.478 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29280.log: /var/log/ceph/ceph-client.admin.33472.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33472.log.gz 2024-04-26T01:26:57.478 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29280.log.gz 2024-04-26T01:26:57.478 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36953.log 2024-04-26T01:26:57.478 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35183.log 2024-04-26T01:26:57.478 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32662.log.gz 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36953.log.gz 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25270.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28848.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35183.log.gz 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.25270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29262.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25270.log.gz 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28848.log.gz 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35994.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29262.log.gz 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35994.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.35994.log.gz --verbose 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-client.admin.36849.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30116.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32782.log 2024-04-26T01:26:57.479 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36849.log.gz 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31812.log 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30116.log.gz 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35602.log 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32782.log.gz 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30630.log 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31812.log.gz 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28879.log 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35602.log.gz 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30630.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.36033.log 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30630.log.gz 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30942.log 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28879.log.gz 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36158.log 2024-04-26T01:26:57.480 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36033.log.gz 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29273.log 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30942.log.gz 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33274.log 0.0% 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36158.log.gz 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.29273.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25073.log 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29273.log.gz 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36032.log 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33274.log.gz 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.25073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31652.log 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25073.log.gz 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34191.log 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36032.log.gz 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30252.log 2024-04-26T01:26:57.481 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31652.log.gz 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:gzip/var/log/ceph/ceph-client.admin.34191.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30912.log 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34191.log.gz 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29828.log 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30252.log.gz 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37187.log 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30912.log.gz/var/log/ceph/ceph-client.admin.29828.log: 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34165.log 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29828.log.gz 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35596.log 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37187.log.gz 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33389.log 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34165.log.gz 2024-04-26T01:26:57.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31172.log 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35596.log.gz 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5/var/log/ceph/ceph-client.admin.33389.log: --verbose -- /var/log/ceph/ceph-client.admin.30532.log 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33389.log.gz 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34456.log 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31172.log.gz 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34217.log 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30532.log.gz 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29276.log 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34456.log.gz 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35216.log 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34217.log.gz 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35218.log 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29276.log.gz 2024-04-26T01:26:57.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32613.log 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35216.log.gz 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33761.log 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35218.log.gz 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34326.log 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32613.log.gz 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33761.log.gz 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33999.log 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34326.log.gz 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28847.log 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36040.log 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33999.log.gz 2024-04-26T01:26:57.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29284.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28847.log.gz 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35647.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36040.log.gz 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31197.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29284.log.gz 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35555.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35647.log.gz 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29872.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31197.log.gz 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31483.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35555.log.gz 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32932.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29872.log.gz 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31443.log 2024-04-26T01:26:57.485 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31483.log.gz 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35600.log 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32932.log.gz 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32768.log 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31443.log.gz 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29704.log 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35600.log.gz 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34430.log 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32768.log.gz 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35613.log 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29704.log.gz 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34430.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.34430.log.gz /var/log/ceph/ceph-client.admin.36744.log 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32219.log 2024-04-26T01:26:57.486 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35613.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35990.log 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36744.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33068.log 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32219.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32936.log 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35990.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28857.log 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33068.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34716.log 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32936.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36875.log 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28857.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34716.log.gz 2024-04-26T01:26:57.487 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35207.log 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36875.log.gz 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30070.log 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32261.log/var/log/ceph/ceph-client.admin.35207.log: 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35207.log.gz 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31896.log 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30070.log.gz 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32562.log 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32261.log.gz 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35635.log 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31896.log.gz 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31736.log 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32562.log.gz 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29869.log 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35635.log.gz 2024-04-26T01:26:57.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34820.log 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31736.log.gz 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37395.log 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29869.log.gz 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35998.log 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34820.log.gz 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37395.log.gz 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31308.log 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28768.log 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35998.log.gz 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35028.log 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31308.log.gz 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34059.log 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28768.log.gz 2024-04-26T01:26:57.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30854.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35028.log.gz 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33171.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34059.log.gz 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37135.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30854.log.gz 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34846.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33171.log.gz 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31786.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37135.log.gz 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33555.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34846.log.gz 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36614.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31786.log.gz 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35469.log 2024-04-26T01:26:57.490 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33555.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37265.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36614.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29876.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35469.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28876.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37265.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35206.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29876.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36458.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28876.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29248.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35206.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31234.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36458.log.gz 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34352.log 2024-04-26T01:26:57.491 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29248.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31147.log 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31234.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34482.log 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34352.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36083.log 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31147.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32053.log 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34482.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31739.log 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36083.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34924.log 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32053.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37161.log 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31739.log.gz 2024-04-26T01:26:57.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34924.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34924.log.gz 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30395.log 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37161.log.gz 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30938.log 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35054.log 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30395.log.gz 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30938.log.gz 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36692.log 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31417.log 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35054.log.gz 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36692.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36692.log.gz -5 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34274.log 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T01:26:57.493 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35579.log 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31417.log.gz 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36432.log 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34274.log.gz 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28870.log 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35579.log.gz 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29730.log 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36432.log.gz 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36075.log 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28870.log.gz 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35638.log 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29730.log.gz 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33446.log 2024-04-26T01:26:57.494 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36075.log.gz 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35106.log 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35638.log.gz 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36536.log 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33446.log.gz 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35106.log.gz 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37343.log 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32362.log 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36536.log.gz 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35186.log 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37343.log.gz 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28877.log 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32362.log.gz 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-04-26T01:26:57.495 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35186.log: /var/log/ceph/ceph-client.admin.28877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32478.log 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28877.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35186.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28836.log 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34950.log 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32478.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33908.log 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28836.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34638.log 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34950.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31626.log 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33908.log: /var/log/ceph/ceph-client.admin.34638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30190.log 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34638.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33908.log.gz 2024-04-26T01:26:57.496 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31626.log.gz 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34560.log 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30190.log.gz 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35191.log 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33821.log 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34560.log.gz 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36927.log 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35191.log.gz 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29678.log 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33821.log.gz 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30506.log 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36927.log.gz 2024-04-26T01:26:57.497 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35533.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29678.log.gz 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28664.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30506.log.gz 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34534.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35533.log.gz 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28664.log.gz 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36796.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34534.log.gz 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33093.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29263.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36796.log.gz 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36979.log 2024-04-26T01:26:57.498 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33093.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29263.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28868.log 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32873.log 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36979.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28868.log: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28868.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37057.log 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28689.log 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32873.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34586.log 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37057.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32077.log 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28689.log.gz 2024-04-26T01:26:57.499 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36142.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34586.log.gz 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30019.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32077.log.gz 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32401.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36142.log.gz 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34508.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30019.log.gz 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32401.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32401.log.gz 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36588.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34508.log.gz 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33912.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37083.log 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36588.log.gz 2024-04-26T01:26:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35217.log 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33912.log.gz 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36106.log 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37083.log.gz 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30829.log 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35217.log: /var/log/ceph/ceph-client.admin.36106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28861.log 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35217.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36106.log.gz 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34898.log 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30829.log.gz 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34742.log 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28861.log.gz 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30589.log 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34898.log.gz 2024-04-26T01:26:57.501 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35211.log 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34742.log.gz 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36113.log 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30589.log.gz 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30746.log 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35211.log.gz 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33967.log 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36113.log.gz 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.30746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33498.log 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30746.log.gz 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28874.log 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33967.log.gz 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35595.log 2024-04-26T01:26:57.502 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33498.log.gz 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24954.log 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28874.log.gz 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28742.log 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35595.log.gz 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.24954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37031.log 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24954.log.gz 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33710.log 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28742.log.gz 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32740.log 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37031.log.gz 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35213.log 2024-04-26T01:26:57.503 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33710.log.gz 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36718.log 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32740.log.gz 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31515.log 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35213.log.gz 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31051.log 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36718.log.gz 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34105.log 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31515.log.gz 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32698.log 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31051.log.gz 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31514.log 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34105.log.gz 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.32698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31357.log 2024-04-26T01:26:57.504 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32698.log.gz 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31514.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31514.log.gz 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35611.log 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31357.log.gz 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33601.log 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35628.log 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35611.log.gz 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33135.log 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33601.log.gz 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34664.log 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35628.log: /var/log/ceph/ceph-client.admin.33135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35614.log 2024-04-26T01:26:57.505 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35628.log.gz 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33135.log.gz 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34378.log 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34664.log.gz 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29652.log 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35614.log: /var/log/ceph/ceph-client.admin.34378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31839.log 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35614.log.gz 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34378.log.gz 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29652.log.gz 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32454.log 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33846.log 2024-04-26T01:26:57.506 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31839.log: /var/log/ceph/ceph-client.admin.32454.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.31839.log.gz --verbose 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-client.admin.28849.log 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32454.log.gz 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33846.log.gz 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31938.log 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37239.log 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.28849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28849.log.gz 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.31938.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37005.log 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31938.log.gz 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37239.log.gz 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28865.log 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36901.log 2024-04-26T01:26:57.507 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37005.log: /var/log/ceph/ceph-client.admin.28865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36770.log 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37005.log.gz 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28865.log.gz 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33686.log 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36901.log.gz 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36770.log.gz 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29281.log 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29222.log 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33686.log: /var/log/ceph/ceph-client.admin.29281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35002.log 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33686.log.gz 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29281.log.gz 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33042.log 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29222.log.gz 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33228.log 2024-04-26T01:26:57.508 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35002.log: /var/log/ceph/ceph-client.admin.33042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29261.log 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33042.log.gz 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35002.log.gz 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.33228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33228.log.gz 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35201.log 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34872.log 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.29261.log: /var/log/ceph/ceph-client.admin.35201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29261.log.gz 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35201.log.gz 2024-04-26T01:26:57.509 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.34872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34872.log.gz 2024-04-26T01:26:57.510 INFO:teuthology.orchestra.run.smithi045.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-26T01:26:57.557 INFO:teuthology.orchestra.run.smithi071.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-26T01:26:57.671 INFO:teuthology.orchestra.run.smithi071.stderr: 90.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-26T01:26:57.690 INFO:teuthology.orchestra.run.smithi181.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-26T01:26:57.720 INFO:teuthology.orchestra.run.smithi071.stderr: 91.0% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-04-26T01:26:57.769 INFO:teuthology.orchestra.run.smithi037.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-26T01:26:57.804 INFO:teuthology.orchestra.run.smithi045.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-04-26T01:26:57.900 INFO:teuthology.orchestra.run.smithi045.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-26T01:26:57.908 INFO:teuthology.orchestra.run.smithi071.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-26T01:26:57.975 INFO:teuthology.orchestra.run.smithi181.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-04-26T01:26:58.091 INFO:teuthology.orchestra.run.smithi071.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-04-26T01:26:58.092 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-26T01:26:58.093 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m0.730s 2024-04-26T01:26:58.093 INFO:teuthology.orchestra.run.smithi071.stderr:user 0m1.931s 2024-04-26T01:26:58.093 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m0.101s 2024-04-26T01:26:58.111 INFO:teuthology.orchestra.run.smithi037.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-04-26T01:26:58.152 INFO:teuthology.orchestra.run.smithi181.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-26T01:26:58.201 INFO:teuthology.orchestra.run.smithi181.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-04-26T01:26:58.203 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-04-26T01:26:58.203 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m0.820s 2024-04-26T01:26:58.203 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m2.374s 2024-04-26T01:26:58.203 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.079s 2024-04-26T01:26:58.227 INFO:teuthology.orchestra.run.smithi037.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-26T01:26:58.283 INFO:teuthology.orchestra.run.smithi037.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-04-26T01:26:58.333 INFO:teuthology.orchestra.run.smithi037.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-26T01:26:58.335 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-04-26T01:26:58.335 INFO:teuthology.orchestra.run.smithi037.stderr:real 0m0.999s 2024-04-26T01:26:58.335 INFO:teuthology.orchestra.run.smithi037.stderr:user 0m3.857s 2024-04-26T01:26:58.335 INFO:teuthology.orchestra.run.smithi037.stderr:sys 0m0.205s 2024-04-26T01:26:58.483 INFO:teuthology.orchestra.run.smithi045.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-26T01:26:58.551 INFO:teuthology.orchestra.run.smithi045.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-04-26T01:26:58.552 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-04-26T01:26:58.552 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m1.208s 2024-04-26T01:26:58.552 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m3.318s 2024-04-26T01:26:58.552 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m0.104s 2024-04-26T01:26:58.553 INFO:tasks.ceph:Archiving logs... 2024-04-26T01:26:58.553 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi037/log 2024-04-26T01:26:58.554 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:26:59.236 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi045/log 2024-04-26T01:26:59.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:26:59.657 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi071/log 2024-04-26T01:26:59.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:26:59.937 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi181/log 2024-04-26T01:26:59.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T01:27:00.290 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T01:27:00.300 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T01:27:00.300 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:27:00.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:27:00.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:27:00.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T01:27:00.425 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T01:27:00.438 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T01:27:00.438 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:27:00.441 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:27:00.443 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:27:00.445 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T01:27:00.452 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:27:00.452 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-04-26T01:27:00.452 INFO:teuthology.orchestra.run.smithi037.stdout:+hv01.front.sepi 67.205.162.81 3 u 21 64 177 0.057 -1.130 0.922 2024-04-26T01:27:00.452 INFO:teuthology.orchestra.run.smithi037.stdout:+hv02.front.sepi 63.231.80.2 3 u 24 64 177 0.079 +2.005 1.322 2024-04-26T01:27:00.452 INFO:teuthology.orchestra.run.smithi037.stdout:*hv03.front.sepi 74.6.168.72 3 u 22 64 177 0.075 +0.369 0.916 2024-04-26T01:27:00.452 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:27:00.455 INFO:teuthology.orchestra.run.smithi045.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:27:00.455 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================== 2024-04-26T01:27:00.455 INFO:teuthology.orchestra.run.smithi045.stdout:+hv01.front.sepi 67.205.162.81 3 u 27 64 177 0.088 -2.096 0.728 2024-04-26T01:27:00.455 INFO:teuthology.orchestra.run.smithi045.stdout:+hv02.front.sepi 63.231.80.2 3 u 22 64 177 0.090 -0.651 1.005 2024-04-26T01:27:00.455 INFO:teuthology.orchestra.run.smithi045.stdout:*hv03.front.sepi 74.6.168.72 3 u 26 64 177 0.090 -0.018 0.795 2024-04-26T01:27:00.456 INFO:teuthology.orchestra.run.smithi045.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:27:00.457 INFO:teuthology.orchestra.run.smithi071.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:27:00.457 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================== 2024-04-26T01:27:00.457 INFO:teuthology.orchestra.run.smithi071.stdout:+hv01.front.sepi 67.205.162.81 3 u 20 64 177 0.069 -1.965 0.856 2024-04-26T01:27:00.457 INFO:teuthology.orchestra.run.smithi071.stdout:*hv02.front.sepi 63.231.80.2 3 u 24 64 177 0.120 +0.419 1.055 2024-04-26T01:27:00.457 INFO:teuthology.orchestra.run.smithi071.stdout:+hv03.front.sepi 74.6.168.72 3 u 18 64 177 0.103 -0.065 0.856 2024-04-26T01:27:00.458 INFO:teuthology.orchestra.run.smithi071.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:27:00.458 INFO:teuthology.orchestra.run.smithi181.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T01:27:00.459 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================== 2024-04-26T01:27:00.459 INFO:teuthology.orchestra.run.smithi181.stdout:+hv01.front.sepi 67.205.162.81 3 u 29 64 177 0.125 -1.818 1.123 2024-04-26T01:27:00.459 INFO:teuthology.orchestra.run.smithi181.stdout:*hv02.front.sepi 63.231.80.2 3 u 22 64 177 0.129 +0.578 1.100 2024-04-26T01:27:00.459 INFO:teuthology.orchestra.run.smithi181.stdout:+hv03.front.sepi 74.6.168.72 3 u 21 64 177 0.098 +0.156 1.130 2024-04-26T01:27:00.459 INFO:teuthology.orchestra.run.smithi181.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T01:27:00.459 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T01:27:00.469 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T01:27:00.469 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T01:27:00.479 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T01:27:00.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T01:27:00.498 INFO:teuthology.task.internal:Duration was 860.184583 seconds 2024-04-26T01:27:00.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T01:27:00.507 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T01:27:00.508 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:27:00.510 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:27:00.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:27:00.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T01:27:00.555 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T01:27:00.555 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-04-26T01:27:00.555 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:27:00.598 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-04-26T01:27:00.598 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:27:00.609 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-04-26T01:27:00.610 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:27:00.619 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-04-26T01:27:00.620 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T01:27:00.630 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T01:27:00.630 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:27:00.643 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:27:00.654 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:27:00.669 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T01:27:00.700 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T01:27:00.701 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:27:00.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:27:00.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:27:00.737 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T01:27:00.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T01:27:00.809 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T01:27:00.809 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:27:00.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:27:00.829 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:27:00.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T01:27:00.852 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T01:27:00.863 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:00.866 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:00.874 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:00.875 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-04-26T01:27:00.881 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-04-26T01:27:00.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:00.894 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-04-26T01:27:00.905 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-04-26T01:27:00.974 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:27:00.979 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:27:00.984 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:27:00.988 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T01:27:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T01:27:01.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T01:27:01.031 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T01:27:01.031 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi037 2024-04-26T01:27:01.032 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:27:01.055 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi045 2024-04-26T01:27:01.056 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:27:01.077 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi071 2024-04-26T01:27:01.078 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:27:01.099 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-25_21:00:03-rados-squid-distro-default-smithi/7674000/remote/smithi181 2024-04-26T01:27:01.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T01:27:01.120 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T01:27:01.120 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:27:01.122 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:27:01.125 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:27:01.141 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T01:27:01.164 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T01:27:01.177 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T01:27:01.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T01:27:01.187 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T01:27:01.187 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:27:01.190 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:27:01.192 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:27:01.194 INFO:teuthology.orchestra.run.smithi037.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:27 /home/ubuntu/cephtest 2024-04-26T01:27:01.194 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T01:27:01.196 INFO:teuthology.orchestra.run.smithi045.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:27 /home/ubuntu/cephtest 2024-04-26T01:27:01.198 INFO:teuthology.orchestra.run.smithi071.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:27 /home/ubuntu/cephtest 2024-04-26T01:27:01.212 INFO:teuthology.orchestra.run.smithi181.stdout: 658615 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 01:27 /home/ubuntu/cephtest 2024-04-26T01:27:01.214 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T01:27:01.225 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T01:27:01.249 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 860.1845834255219 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-26T01:27:01.249 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T01:27:01.328 INFO:teuthology.run:pass